On her Substack, she later wrote a follow-up piece where she walked through some of the rebuttals — and the rebuttals to the rebuttals. One of the most prominent rebuttals to an argument like that is the good old Jevons paradox: when the cost of a desirable good goes down, the demand for it skyrockets. So even if artificial intelligence becomes much cheaper, you’ll just have more and more demand for it, and more and more jobs. Sun’s rebuttal to that rebuttal is that, sure, you might have more demand for AI, but that doesn’t necessarily mean the demand gets fulfilled by humans. I think that’s the crux of the argument.

The Complexity Ratchet

There’s one part of this argument that often gets missed. I don’t know if it has a name — I like to call it the complexity ratchet — and I think it’s what might still end up saving humans.

If you look at the tech tree of civilization, it’s one long walk up the ladder of complexity: from mud huts to wooden buildings to concrete buildings to skyscrapers, or from wheels to cars to jet airplanes to spaceships and rockets. Each of these is orders of magnitude more complex than the one before it.

Why Spreadsheets Didn’t Kill Accounting

A very clean, recent example — one within most of our lifetimes — is the advent of the personal computer. The killer app for the personal computer was the spreadsheet. People didn’t really know what to do with a personal computer until VisiCalc came around and they saw spreadsheets. That’s what made everyone realize what computers could actually be used for.

When spreadsheets came out, the overwhelming lament at the time was that they would wipe out the accounting profession. Decades later, we know they didn’t. Quite the contrary — they probably 10x’d or 100x’d the number of accountants.

Why? The complexity ratchet. Because spreadsheets made it so easy to do more and more complex financial engineering, the result was that you needed more and more human accountants to stay on top of all that complex financial engineering. You can make the same argument for most digital technology. I don’t think you’d have laws and bills in Congress that run to thousands of pages if you didn’t have word processors.

How Technology Raises Civilization’s Complexity

So the complexity ratchet argument is that all of these technologies irreversibly ratchet up the level of complexity in our civilization. They let us create, manage, and live with more and more complex legal, societal, financial, and educational structures. And those structures just become part of our lives, part of our civilization.

Even though you need machines to manipulate a lot of these systems, you ultimately also need a lot of humans. The financial system is a great example: you need a lot of complex machinery to operate today’s financial markets — and not just the markets, but all the structures around them: companies, LLCs, C corps, S corps, public and private corporations, shareholder and ownership structures. You created a lot of complexity, and you did so irreversibly. You needed machines to climb that complexity ladder, and you need machines to maintain civilization at that level of complexity. But all the while, you didn’t do away with humans — just as with spreadsheets, you didn’t reduce the total number of accountants in the world.

So I think the complexity ratchet might be a corollary of — maybe a necessary companion to — the now-familiar Jevons paradox argument.

Will AI Make Us Accountants or Farmers?

Now, there’s a rebuttal to this too: okay, AI will enable a massive leap in the complexity of the structures we build and use, but sure, the AI itself will manage all of it for you. I think that’s still plausible at this point. I don’t think we know enough yet to say whether it goes the way it did for accountants — where you still needed a lot of accountants — or whether it goes the way of agriculture, where we still produce more than enough food for the entire planet but the number of farmers, in both absolute and relative terms, completely fell off. We just don’t need many farmers anymore.

So will all this go the way of accountants, or the way of farmers? You can draw some historical parallels here. Farming is a very physical activity — it’s from the world of atoms, not the world of bits. There’s some ceiling, some limit, to how much food you need or want, and how much food you can store and move around. Accounting and law and finance, on the other hand, are purely abstract, completely conceptual. There’s no ceiling to the complexity of financial structures, as long as someone is still getting some utility out of it.

If you believe that argument, then even with AI — even with the complexity ratchet enabled by AI — you’re still going to need a lot of humans in the future to stay on top of all that complexity. In other words, we go the way of the accountants rather than the farmers.

Only time will tell which way we end up going. But I wanted to talk about the complexity ratchet argument, because the Jevons paradox argument is pretty well known by now, and I haven’t seen the complexity ratchet argument made — at least not to my knowledge.

All the best!

That, I think, is how you’d have to summarize the state of the software engineering job market right now. Everyone keeps saying AI is eating the entry-level rungs of the ladder. (I said it back in early 2024!) And to show just how confusing things are, I want to put two seemingly contradictory pieces of data side by side.

Two Graphs, Two Narratives

The first graph comes from a report from Citadel Securities that came out about a week ago. The author’s argument is that AI-exposed industries are not seeing headcount contraction. Software engineering job postings have been spiking since around mid-2025. He looks at another heavily AI-exposed industry — customer service — and sees the same upward trend in postings. The thesis: AI is creating jobs and accelerating new business formation, especially in industries exposed to AI.

The second graph is from a paper by Brynjolfsson et al., cited in the recently released Stanford AI Index 2026 — specifically Chapter 4 on the economy. The Brynjolfsson paper looks at the number of people actually employed in these roles. And the picture is starkly different: among workers ages 22 to 25, employment in the most AI-exposed occupations has fallen roughly 16% relative to the least exposed. Junior software engineers, in particular, have fallen off a cliff since around late 2022 — with no comparable decline for mid-career and senior engineers.

So which is it? Is AI creating jobs, or destroying entry-level jobs?

The simplest reconciliation is that these two graphs are measuring different things.

The Citadel chart counts job postings. That’s a forward-looking signal: how many roles are open, how many companies are willing to hire, what the demand side intends to do over the next few months. It’s not a snapshot of who is currently employed.

The Brynjolfsson chart counts employed people. That’s a backward-looking signal: who actually has a job right now. The two can diverge for long stretches, especially in industries going through structural change.

So both can be true at the same time. The pipeline of new openings can be filling up while the existing population of junior engineers has shrunk. One is the leading indicator. The other is the lagging one.

The Causality Is Messy

There’s a wrinkle, though. The Brynjolfsson authors issued an update to their paper where, after controlling for various confounders, they found the timing of the employment decline shifts to 2024 rather than late 2022. And the causality gets a lot messier. You can’t pin it just on AI. You can’t rule out the usual culprits — ZIRP and massive hiring during COVID.

The honest read on the data is: yes, junior software engineering employment is down. No, we don’t yet have a clean attribution to AI. Most likely it’s a stew — AI plus the COVID overhiring correction plus the rate environment.

Ground Truth

I talk to a lot of folks who are close to graduation, or have recently graduated, or are trying to break in. The vibe matches the Brynjolfsson chart, not the Citadel chart. The entry-level software engineering market just feels really tough right now.

But the Citadel chart is forward-looking. So if you take it seriously, the tide may have already turned. The worst may be behind us.

The Old Scripts Are Torn Up

The framing I keep coming back to when I talk to people about this: there is plenty of opportunity. But all the old scripts have been torn up.

For more than the last two decades — through the internet boom and into the cloud era — there were fairly stable scripts for a software engineering career. Stable job expectations. A stable ladder defining how to grow your skills and climb through the levels. A well-known script for what to study, what to major in, how to interview.

AI, especially at the junior end, is completely rewriting what that ladder means. The well-understood scripts that an entire generation of new grads, engineering managers, and leaders have been running on no longer work. They have to be rewritten, by all of us, while we’re already running.

So if you’re trying to start a career in software right now, or trying to figure out how to manage one, the right mental model is to… figure it out yourself, because nobody else has any idea either.

There is plenty of opportunity. The old scripts just don’t work anymore. Write your own.

All the best!

Here are some observations. I can’t quite thread them into a coherent narrative. The crystal ball is cloudy, and also the clouds are moving very fast.

• every AI startup wants to be a pure product, almost SaaS-like company. Almost all of them end up becoming bespoke solution builders, aka consultants. Nobody likes being called that, so they say “forward deployed engineer”. You want to build agents for companies, and every company is a snowflake.
• Palantir looked like a consultancy for a decade before it dawned on everyone that they had a durable product behind it. The startups are hoping to reproduce that playbook: structure the engagements to harvest transferable artifacts rather than bespoke deliverables.
• The major model vendors are moving aggressively into verticals. See recent product pushes such as “Claude for finance/law/healthcare”. So much for the 2023-25 thesis of finding shelter in vertical specialization.
• The major model vendors are aggressively hiring FDEs. FDEs are subsidized by the token business. They don’t have to pay for themselves — they exist to expand usage. A startup’s FDEs have to be explicitly funded. The vertical startup is competing with a cross-subsidy they can’t match.
• Capability is not access. Can the major model vendors really shmooze enterprise sales? Or: rizz is the only moat.
• Speed of erosion matters. If a startup can carve a niche that lasts 6 months before the giants catch up, maybe it can convert that into a trusted relationship that persists. In other words, move fast enough to turn a perishable asset into a durable one.
• maybe the very framing of “having a moat” is incorrect. When the ground under your feet shifts so fast, you probably want some notion of “velocity + customer intimacy”.

I’ve been building in this space for a while and I have some thoughts that I’m hoping will clear some of these things up. Let’s start with skills. What are skills really for?

Skills are desire paths

Skills have been a formalization of something people were already doing. That’s why I call them a desire path — you look for where people are walking, where the grass is worn out, and that’s where you build the sidewalks.

Before skills existed as a spec, people were stuffing additional context into markdown files and feeding them to the model through whatever harness or agent they were using. Anthropic formalized that into a proper spec, while adding context-saving mechanics like progressive disclosure.

The whole point of skills is to articulate either procedural knowledge or informational knowledge. Procedural knowledge is a set of instructions for how to do something — you already know what to do, and you want to simply tell the model. You don’t want the model to have to plan or think or reason through a process that is largely pre-determined. Informational knowledge is general domain context that helps the model on a task.

To take a common example: if you were building a data analysis agent that queries a bunch of tables, you’d give it knowledge about what the column names mean inside your company, because those meanings are specific to your domain. That’s informational knowledge. But you’d also give it very specific instructions for how to, say, close the books at the end of the month. That’s not something you want the agent to plan or reason about — you just spell out your process. That’s a procedural skill.

The important thing about skills is that they’re almost always something that is not in world knowledge. It’s additional domain information — whether informational or procedural — that you are making explicit.

SkillsBench empirically showed that curated skills authored by humans bring a huge jump in task performance. Whereas self-generated skills — skills generated by the model — bring no benefit or in some cases even hurt performance. Which makes sense if you follow the logic above: skills are supposed to be something outside the model’s world knowledge. You don’t want the model to generate skills for you. You want to tell it all your knowledge.

A great example is Anthropic’s own PDF skill. They spell out common PDF operations — merging, splitting, extracting — and specify to use PyPDF. The model doesn’t have to go hunt for PDF libraries or plan and reason about well-understood operations. The skill just says: here’s how to do it.

MCP

MCP has been a smashing success, so most people now have a reasonable understanding of it. The core idea is to bring context to the model. And once again, like the skills desire path, MCP is a formalization of function calling. Before MCP, each model provider — OpenAI, Gemini, Anthropic — had their own slightly different syntax and API. MCP formalized that into a spec.

The other thing MCP does, and this is less known, is serve up resources — HTML pages, JavaScript, CSS. This is now the basic mechanism behind MCP apps and ChatGPT apps: they are MCP servers that serve UI via resources.

So why did MCP get a bad reputation? Two technical reasons.

Context hogging. When an MCP server had a large list of tools, just loading it would eat a chunk of your context window. The most well-known culprit was the GitHub MCP server with something like 70 tools — loading it consumed roughly a third of your context window. I’d argue that’s not MCP’s fault. That’s bad MCP server design. If you’re building an MCP server, you probably should not make it a kitchen sink.

Context dumping. When you call an MCP tool, the results land in the window. Unfortunately, a bunch of badly designed servers would produce very large tool results, often larger than they needed to be.

Fortunately, both of these problems have been largely mitigated today by code mode. The idea was initially proposed by Cloudflare, but Anthropic later adopted it as the canonical way to invoke MCP tools. The basic idea is that instead of the model directly getting all tool results, you write a small amount of code to make the tool call and interpret the results. Doing it in code shields the model from both problems.

So I think the initial reasons MCP got a bad rap have been largely addressed. A lot of it was just bad server design, and code mode fixes the underlying mechanism.

Where do CLIs work really well?

That brings us to CLIs. Peter Steinberger famously said he prefers CLIs over MCP. So let’s look at where CLIs actually work well.

They work really well on what I call personal agents. These are agents running on my own machine — a Mac Mini, a VM, whatever — where I control the environment. I’m signed into all the CLIs — GitHub, Google Cloud, Cloudflare — and my agents can just invoke them on my behalf. The auth story on personal machines is clean and seamless.

The other advantage is context efficiency. Agents can do agentic search over a CLI’s function space by invoking --help and walking the subcommand tree. Most modern CLIs are structured as a top-level command with subcommands, each handling a subset of functionality. Agents are surprisingly good at this tree-based exploration. They don’t suffer from the context hogging problem of loading dozens of MCP tool descriptions at once.

Where MCP wins

But CLIs don’t work everywhere. In enterprise scenarios, you really do want MCP.

When you’re doing a proper production deployment, you’re deploying your own binary. You’re not also deploying a bunch of CLIs with some auth credentials alongside your application. You want proper service separation.

An organizational pattern I see across enterprise deployments: one team stands up the official MCP servers for the entire company, and other teams build agents on top. This buys you central governance — one official source of truth for the underlying data. You don’t want people spinning up side MCP servers all over the place, just like in the old days you didn’t want everyone creating side tables and fragmenting the source of truth.

It also buys you separation of concerns. The teams building agents don’t need to worry about standing up MCP servers, and the team building MCP servers doesn’t need to know about every agent use case across the company.

It’s not either/or

The takeaway: these are complementary tools, working at different layers of the agent stack, for different environments, not competing alternatives. Skills encode domain knowledge the model doesn’t have. CLIs work great for personal agents on machines you control. MCP is the right abstraction for enterprise deployment with governance and service separation.

I’ve been a heavy user of agents for a while now — not just coding agents, but agents in general, for all my work. And I’m beginning to bump up against some walls when it comes to human-agent interaction. I want to share some thoughts on what those walls are and what might actually solve them.

The Chat UI Has a Low Ceiling

Look at the direction agent UI is going. Cursor 3.0 just shipped, and its big selling point is managing tons of agents — local or cloud, running in parallel — all in one UI. Claude Code looks similar. And it’s not just the agents: a ton of SaaS products (Linear, PostHog, Attio) have converged on the same ChatGPT-like interface: chat front and center, a sidebar of previous conversations.

This kind of UI has a low floor — it makes it easy to get started with agents. But I’d claim it also has a fairly low ceiling. Once you do a lot of your work (or even all your work) through agents, you hit a wall.

The wall is sensemaking — making sense of all that work after it’s done.

Narrating Your Work

For the longest time, I’ve been a fan of this concept called narrating your work. The idea is simple: you keep synthesized notes of the things you’ve been working on. A work log.

I was first exposed to this at Google, where engineers posted weekly “snippets” — short updates on what they’d done that week. Most people dropped a few bullet points. But every once in a while, there were those three or four engineers (you know who you are) who wrote mini essays in their snippets. Subjective, synthesized views of what they were working on, what was working, what wasn’t, what their plans were.

I got enormous value from reading those. These were engineers in distant parts of the company — it was one-sided in the sense that I never worked with them or met them in person. But reading their essays gave me a deep insight into the kinds of work they were doing. It made me feel a kinship with them. And it’s what encouraged me to keep up my own snippet-writing habit.

The primary audience is yourself — reflection over the mountain of work — but the secondary audience is everyone else who stumbles across it.

The Sensemaking Problem

Here’s the point: all these agent UIs make it really difficult to make sense of all the work you’ve done with your agents. You cover ten miles in a day where you would have covered one without agents. But at the end of the day — or the week — you’re left disoriented. You’ve done so much work. But if you try to think back: What did I actually accomplish? What’s the higher-level picture? It’s hard to reconstruct.

You need that synthesis so you can learn from it and plan your next chunk of work. And that, I think, is the main shortcoming of current agent UIs. They let you do a ton of work, but leave you in a disoriented place. Synthesis over all that work is an unsolved problem.

Agent Kitchen: A First Attempt

So I built a tool called Agent Kitchen that gives me a higher-level view of all the work I’ve been doing with my agents — across repos, across my home directory, across random directories on my computer. I wrote about the motivation in a previous post.

I’ve been using it daily since I released it recently. It’s now my primary interface into all my Claude Code sessions. I get a global view of sessions across different repos and directories, and I can click to resume any of them.

But even with this, I want more. This still isn’t quite the high-level synthesized narrative I’m after. Ideally, I’d want a paragraph or two of plain natural language at the top — a synthesis of what I got done in a given repo or directory, with links off to the relevant sessions so I can click to resume them.

Another common annoyance: agent chats grow really long, with many turns between you and the agent, and you just have to scroll through them. There’s no minimap, no bookmarks to just your inputs — nothing like the minimap in VS Code that gives you a global view of a long file. These are small UI issues, but they add up to the same problem: agents treat everything as a wall of text.

UX Is Now the Bottleneck

Overall, I think we’re arriving at a place where the UI/UX of using agents is becoming the bottleneck in how effective we can be with them. The models are getting better. The harnesses and agents are getting better. But I believe we’re now bottlenecked by the UX paradigm.

We spent the last two years making agents that can do more work. Now we need interfaces that help humans understand all that work. Until then, we’re sprinting blind.

Autoresearch

Andrej Karpathy’s AutoResearch showed a clean version of this idea. Take a program that trains a small GPT model, and let a coding agent edit it directly — making changes to architecture, hyperparameters, optimizer, anything — then running a training job and checking whether a performance metric improved. If it did, keep the change. If not, git-reset back to the last good version. Greedy hill-climbing, running overnight automatically.

A human ML researcher performs the same loop manually: set off a training run, check metrics, tweak the training logic, repeat. AutoResearch automates that entire cycle. The crucial enabler is having one very unambiguous, very clear metric to optimize. The agent does the hill-climbing all by itself.

MetaHarness

Meta-Harness takes this idea and applies it to model harnesses — what we used to call scaffolding. A harness is all the logic you wrap around a base model to make it accomplish a task: the reasoning loop, tool use, planning, prompt engineering. Claude Code is a harness, just a very general-purpose one. So is any agent framework you build for a specific purpose.

Meta-Harness is trying to do for harnesses what Autoresearch did for training: can you automatically construct the optimal harness for a given task?

The architecture is simple, and mirrors what a human AI engineer does. A coding agent — Claude Code with Opus 4.6, in their experiments — acts as the proposer: it creates a new harness, which gets evaluated on a benchmark. The scores, the harness source code, and crucially all the execution traces (prompts, tool calls, model outputs) get stored back to the file system. Then the agent loops back.

There is no complex co-ordination mechanism. It is just the file system, folders full of harness code, benchmark results, and traces. The proposer reads everything from previous iterations via standard tools (grep, cat, ls), does failure analysis on the traces, and proposes something better. There’s a minimal domain-specific prompt that tells the agent where to write harnesses and what files it can modify, but beyond that, no special-purpose programmatic logic drives the improvement. The general-purpose proposer agent (Claude Code, in this case) is smart enough to poke around the folders and files from previous runs, analyze where it failed, propose improvements, and keep the loop going.

And it works. They tested across multiple domains — text classification, math reasoning (IMO-level problems), and agentic coding (TerminalBench-2) — and the meta-harness loop produced harnesses that beat the strongest handcrafted baselines in each.

This is very bitter-lesson-pilled: throw search and compute at the problem, and you’ll outperform bespoke human engineering.

AutoResearch operates on a narrow signal. It only works with the last good train.py (reverting on failure) and one scalar metric. Meta-Harness operates on a much richer information set: the source code, evaluation scores, and full execution traces from every previous attempt are preserved and accessible. That richer context gives the proposer far more to work with when diagnosing failures and proposing improvements.

I Did This By Hand (And Regretted It)

This resonates for me personally. I recently designed and open-sourced Proceda, a harness for automating standard operating procedures into agents, and spent days benchmaxing it on SOP-Bench. My loop looked exactly like Meta-Harness — run the benchmark, look at the score, ask Claude Code to analyze the failure traces, improve the harness, repeat — except I was the one closing the loop manually.

I got to SOTA on SOP-Bench doing this, but at the end I was kicking myself for not setting up an AutoResearch-style automated loop. Which is precisely what Meta-Harness formalizes.

The Takeaway

The main thrust of the entire agent field right now is removing humans from agentic loops. AutoResearch and Meta-Harness are two concrete illustrations of how to do it: define a clear metric, give the agent access to its own history, and let it rip.

For me, this was a notification at 5:30 pm on weekdays to get moving to pick up kids from daycare before it closed at 6 pm. It understood this about my life with very high confidence after maybe a week of silent observation.

I separately looked up my location history and noticed how much it was like clockwork. A perfect triangle between home, daycare and the office. Clockwise in the morning, counter-clockwise in the evening, in time windows varying by a few minutes.

I remember thinking to myself: the bulk of my life can be captured in—and predicted by—a protobuf that is probably under a kilobyte. On a personal level this gave me a lot of food for thought.

But on a philosophical level it drove home another point: one way to look at agency or free will is the ability to surprise a predictive model.

This means that a huge amount of my daily work—the conclusions I’ve reached, the investigations I’ve done, the decisions I’ve made—lives scattered across hundreds of agent sessions. And unless I do something about it, all of that just… disappears. It’s the browser history problem all over again.

The Browser History Problem, Redux

Browser history has never been good. You visit a page, read something useful, close the tab, and it’s gone. You vaguely remember that you saw something relevant three days ago, but good luck finding it.

That’s exactly where we are with coding agent sessions right now. Every Claude Code or Codex session is a rich conversation full of context, decisions, and insights. And when you’re done, it gets serialized into a JSONL file in a hidden directory and effectively ceases to exist.

I have months of these sessions. They contain things I’ve figured out about my own codebase, debugging insights, architectural decisions, research results. It’s a gold mine that I can’t access.

Why Existing Tools Didn’t Fit

There’s no shortage of agent orchestration tools. Sculptor, Conductor, Claude Squad—they’re all good projects with thoughtful design choices. But they all share an assumption that doesn’t match how I work: you start by picking a single git repo.

That’s not how I use coding agents. My Claude Code sessions start at my home directory. I have a global store of memories, skills, and workflows in ~/AGENT that I want applied everywhere, regardless of which repo I’m in. And my work routinely spans multiple repositories—a feature implementation might touch three or four small repos that are all connected. The single-repo model is a straitjacket.

The other mismatch is about what I actually need. These tools focus on orchestration—spinning up isolated environments, managing concurrent agents, containerizing for security. Those are real concerns. But they’re not my concern. What I need is sense-making. I need to look back over my last week of agent sessions and understand what happened. I need a high-level timeline that says “Monday: debugging the auth flow. Tuesday-Wednesday: refactored the scanner. Thursday: three separate research sessions on MCP servers.”

Orchestration is a nice-to-have. Understanding your own history is a must-have.

Agent Kitchen

So I built Agent Kitchen. It’s a local web dashboard that reads your Claude Code and Codex session history, generates LLM-powered summaries, and presents everything as a browsable, searchable timeline grouped by repository.

The core workflow is simple. agent-kitchen index scans your sessions and generates summaries (using Claude Haiku via the Agent SDK, so it runs on your Claude Max subscription—no separate API bill). agent-kitchen web serves the dashboard. That’s it.

What you get is a reverse-chronological view of every folder and repo you’ve worked in, with your sessions grouped and summarized. Each repo group has a timeline at the top—an LLM-generated narrative of what you’ve been doing over the past days and weeks. You can click any session to resume it—Agent Kitchen spins up a terminal emulator right there in the browser. You can start new sessions with one click. There’s a time slider to filter by recency, fuzzy search, and vim-style keyboard navigation.

What This Is Really About

Agent Kitchen is a small tool that scratches a specific itch. But the itch points at something bigger.

As coding agents become our primary work interface, the history of our agent sessions becomes the history of our work. It’s not just conversation logs—it’s an institutional record of decisions, investigations, dead ends, and breakthroughs. We’re generating this record every day, and right now we’re throwing almost all of it away.

Browser history became genuinely useful when someone realized it wasn’t just a list of URLs—it was a map of your information-seeking behavior. Agent session history has the same potential. It’s not just a list of conversations. It’s a searchable, summarizable record of everything you’ve figured out.

The tools we build around agent sessions shouldn’t just help us run more agents in parallel. They should help us understand what we’ve already done. That’s the harder problem, and the more valuable one.

Try It

Agent Kitchen is open source and on PyPI. You can run it without installing anything: uvx agent-kitchen web.

One tip: by default, Claude Code deletes sessions older than 30 days. Set cleanupPeriodDays to something large in your ~/.claude/settings.json to keep them around. Your old sessions have more value than you think.

mcpknife boot --prompt "Free Dictionary API https://dictionaryapi.dev/" \
  | mcpknife mod --prompt "Create one tool: given a word, return synonyms and antonyms" \
  | mcpknife ui

Three stages, composed with Unix pipes. The first reads API docs and generates an MCP server. The second transforms its tools with a natural language prompt. The third generates an interactive web UI. The output is a running MCP app you can use in ChatGPT or any MCP host — built entirely from the command line.

MCP Knife is open source (Apache 2.0) and available on npm. If you want to understand how each piece works and what’s happening under the hood, keep reading.

Three Subcommands, One Pipeline

MCP Knife has three subcommands, each handling one stage of the journey:

mcpknife boot

Given a natural language prompt and some API documentation, boot stands up an entirely new MCP server. Under the hood, this uses MCP Boot, which works in two phases: at startup, an LLM reads your prompt and API docs, plans which tools to create, and generates JavaScript handler functions for each one. At runtime, those cached handlers execute in a sandboxed Node.js VM — no further LLM calls, no per-invocation costs.

For example:

mcpknife boot --prompt "Free Dictionary API https://dictionaryapi.dev/"

That’s it. You now have a running MCP server wrapping the dictionary API.

mcpknife mod

The server from boot might expose more tools than you need, or you might want to reshape them. The mod subcommand transforms tools on an existing MCP server using natural language. It’s built on MCP Blox, which acts as a programmable proxy — it intercepts tool definitions, uses an LLM to generate transformation code, and runs that code in a sandbox.

You can rename tools, hide ones you don’t want, reformat outputs, or compose multiple upstream tools into a single new synthetic tool. All with a prompt:

mcpknife mod --prompt "Create one tool: given a word, return its synonyms and antonyms"

mcpknife ui

Finally, ui auto-generates an interactive web interface for your MCP server. This is powered by MCP Generative UI, a proxy layer that discovers your server’s tools and uses an LLM to create self-contained HTML interfaces for each one. It supports both the MCP Apps standard and the OpenAI Apps SDK (for ChatGPT integration).

mcpknife ui

No prompt needed — the default UI is usually good enough. But if you want to customize, you can pass a styling prompt, or use the built-in _ui_refine tool to iteratively improve the generated UI with natural language feedback.

Composability

You can compose the above invocations using good old Unix pipes. Each stage reads an upstream server URL from stdin and outputs its own URL to stdout. MCP Knife itself is a lightweight process multiplexer, not a monolith. The OS kernel handles the pipes, signal propagation (Ctrl+C, exit codes) flows cleanly through the chain, and each stage is independently replaceable.

A typical pipeline will start with boot, have a few mods in the middle, and end with ui.

You can also use each subcommand standalone. Want to add a UI to an existing MCP server? Just run mcpknife ui --upstream "npx some-mcp-server". Want to reshape tools without changing anything else? Pipe into mcpknife mod. The pieces work together or apart.

Live Example: Dictionary Synonyms App

In the full demo video below, I walk through this exact pipeline with the Free Dictionary API. The three-stage pipeline produces an MCP app that takes a word and displays its synonyms and antonyms in a clean, generated UI — no hand-coded frontend.

And if the UI isn’t quite right? You can refine it. The _ui_refine tool accepts natural language feedback like “show synonyms and antonyms in two columns side by side” and regenerates the interface. You keep iterating until you’re satisfied.

Configuration

All the LLM-powered stages need an API key and model specification. Rather than passing these on every command, you can create a ~/.mcpkniferc config file:

{
  "provider": "anthropic",
  "model": "claude-sonnet-4-20250514",
  "apiKey": "sk-ant-..."
}

CLI flags override config, and project-level .mcpkniferc files override the global one.

Installation

npm install -g mcpknife

Requires Node.js 18+. The repo is open source under the Apache 2.0 license.

Underlying tools

MCP Knife is the culmination of three separate tools I built over the past few weeks:

• MCP Boot — Bootstrap MCP servers from API docs
• MCP Blox — Transform and compose MCP server tools
• MCP Generative UI — Add dynamic UIs to MCP servers

Each of these works standalone, but MCP Knife makes them composable in a way that feels natural on the command line. I’d love to hear your feedback — try it out, let me know what you build, and open issues on the GitHub repo if you run into anything.

I’ve been building and using MCP servers for months now, and a recurring frustration finally pushed me to build something: I kept wanting to reshape MCP servers without modifying them. Rename a tool. Hide the ones I don’t need. Combine two tools into one. Simple transformations that shouldn’t require forking a repo or writing adapter code.

So I built mcpblox, a programmable MCP proxy that takes any existing MCP server and a natural language transform prompt, and produces a new MCP server with reshaped tools. It’s on npm and Apache 2.0 licensed.

(I really wanted the name “mcpipes” but it was taken. “mcp-lego” would have been… legally interesting.)

The Problem

Most MCP servers are designed for general use. They expose a lot of tools, often with verbose names and schemas that don’t quite fit your specific workflow. You end up with tools you don’t need cluttering your agent’s context, naming conventions that don’t match your preferences, and no way to combine multiple upstream tools into the single composite operation you actually want.

The traditional fix is to write a wrapper server by hand. That works, but it’s tedious, fragile, and you lose the ability to iterate quickly. What I wanted was something more like the Unix philosophy: small, composable transformations you can chain together.

How It Works

The core idea is simple. mcpblox sits between your MCP host and an upstream MCP server. At startup, it:

1. Connects to the upstream server and discovers all its tools
2. Sends your natural language transform prompt plus the tool definitions to an LLM
3. The LLM produces a structured transform plan—which tools to rename, modify, hide, or compose into new synthetic tools
4. For each transform, the LLM generates JavaScript functions that run in a sandboxed vm context
5. The result is cached, so subsequent startups with the same prompt skip the LLM entirely

At runtime, tool calls flow through the transform pipeline. Input arguments get transformed, the upstream tool is called, and the output is transformed before returning to the host. Pass-through tools are proxied directly with zero overhead.

A Simple Example

Take the yfinance MCP server—a great tool for financial data that exposes several tools, all prefixed with yfinance_. To strip that prefix:

npx mcpblox \
  --upstream "uvx mcp-server-yfinance" \
  --prompt "Remove the yfinance_ prefix from all tool names" \
  --api-key $ANTHROPIC_API_KEY

That’s it. You now have a proxy MCP server where yfinance_get_price is just get_price. Point any MCP host at http://localhost:8000/mcp and you’re done.

The Real Power: Composition via Unix Pipes

A single transformation is useful. But the real power comes from chaining them—just like Unix pipes.

When mcpblox detects its stdout is a pipe, it binds to an OS-assigned port and writes its URL to stdout. The next mcpblox instance reads that URL from stdin as its upstream. No explicit --upstream flag needed.

mcpblox --upstream "uvx mcp-server-yfinance" \
    --prompt "Remove the yfinance_ prefix from all tool names" \
  | mcpblox \
    --prompt "Create a new tool called compare_stocks that takes two tickers and returns price, market cap, and 52-week range for both"

The first stage strips the prefix. The second stage creates an entirely new synthetic tool that composes multiple upstream calls. Each stage is cached independently—change the second prompt and only the second stage regenerates.

This lets you build up elaborate transformations from simple, understandable steps. The composition model is the same one that makes Unix powerful: complex behavior from simple, composable parts.

Synthetic Tools

This is where things get interesting. The --prompt isn’t limited to renaming or hiding. You can ask mcpblox to create entirely new tools that orchestrate multiple upstream tool calls.

For example, a prompt like:

Create a tool called get_period_returns that calls get_price_history for 1-month, 3-month, 6-month, and 12-month periods, and returns calculated returns for a given ticker.

This generates a synthetic tool with its own schema and an orchestration function that calls the upstream tool multiple times, aggregates the results, and returns a clean response—all from a single natural language description.

Slap a UI on It

In my previous work on generative UI for MCP, I made mcp-generative-ui pipe-friendly too, so you can cap off a pipeline with a UI:

mcpblox --upstream "uvx mcp-server-yfinance" \
    --prompt "Remove yfinance_ prefix" \
  | mcpblox \
    --prompt "Create compare_stocks tool; hide all tools except compare_stocks and get_price_history" \
  | mcp-gen-ui

The result: a base MCP server, transformed and composed through two pipeline stages, with a generated interactive UI on top. The compare_stocks tool renders a graphical comparison instead of raw JSON. All from a shell one-liner with good old Unix pipes.

Try It

npx mcpblox --help

The repo is open and Apache 2.0 licensed. I’d love to hear what transformations and compositions you come up with.

I want to lay out how I see these pieces fitting together. But first, a crucial distinction.

For each of these capabilities, there are multiple competing standards and implementations. It’s important to separate the abstract idea (what capability does this provide?) from the concrete standard (which protocol or standard implements it?). The abstract ideas are what matter for architectural thinking. The concrete implementations will evolve and compete.

Here are the three key abstractions:

1. MCP: Standardized Access to Data and APIs

The first foundational abstraction is MCP (Model Context Protocol). Its key function: standardize how models and agents access data and APIs, without building a forest of bespoke integrations.

Before MCP, every agent needed custom code to talk to every data source. MCP gives you a standard interface—a universal adapter between agents and the world’s APIs and databases. It’s been widely adopted and is, by now, a proven success.

2. Skills: Encoding Domain Knowledge for Agents

The second abstraction is skills. At a high level, skills let you encode instructions, SOPs (standard operating procedures), or domain knowledge in a form that agents can access and follow.

This is critical because not every agent task should involve dynamic planning from scratch. Many workflows are already well-understood. You know the right sequence of steps. You don’t want the agent to improvise—you want it to execute a known procedure reliably.

Skills are the abstraction that captures this. (For a deeper dive into how skills, commands, and sub-agents relate in practice, see this breakdown.) And importantly, skills can use MCPs to access data and APIs, but do so at a higher level of abstraction. For example, a skill might say: “Get the last 30 days of sales for Product X.” That’s a high-level, abstract instruction. Underneath, it gets translated into: which database, which table, which columns, what SQL query—and eventually, an MCP call that sits between the agent and your database.

The separation of concerns is clean: skills encode the what, MCPs handle the how.

3. Generative UI: The Last Mile for Humans

The third and most recent abstraction is putting UIs on top of agents. There are a couple of competing standards here—OpenAI’s Apps SDK and MCP Apps (an extension of MCP supported by multiple vendors)—but the abstract capability is the same: AI agents or MCP servers can now serve up a UI, not just text or tool responses.

UI is a last-mile problem. Agents themselves are far more effective working through APIs, MCPs, and command-line interfaces. UI exists for humans—to consume the final output of an agent’s work. And if UI is just the last-mile output for human consumption, then a necessary implication is that UI must be generative.

Think about it: if an agent executes a bespoke query or a multi-step skill, the result contains a unique mix of personal data, query-dependent information, and even query-independent signals (like PageRank was for Google search results). A static, pre-designed UI can’t anticipate this. The UI needs to be generated dynamically to fit the specific intent in the user’s request.

This is analogous to how search results pages or LLM chat interfaces work—every response is different, and the presentation adapts to the content. Handcrafted, static UIs designed ahead of time simply don’t work when the output is this dynamic.

How They Click Together

Here’s the full picture:

1. Skills encode the workflow—the high-level instructions and domain knowledge.
2. MCP provides standardized access to the data and APIs that skills need.
3. Generative UI renders the final output for human consumption.

A concrete example: An agent executes a skill that says “Get last 30 days of sales for Product X.” The skill translates this into MCP calls to query the right database. The results come back, and an MCP App generates a UI—maybe a chart, a summary table, and some key insights—tailored to this specific result.

Each of these abstractions was a missing piece that, until recently, you had to cobble together from scratch. Now they exist as standardized capabilities. Together they enable agents that can execute real, end-to-end business processes.

BUT… I have to acknowledge the valley of despair and darkness I went through, and that many software engineers are going through now. A lot of those feelings are manifesting in the anti-AI sentiment that now is a common strain. “My handwritten code is better.” “AI code is slop”. “It is a slot machine”.

When ChatGPT came out I had been a professional software engineer for more than two decades. I had built up an entire identity around my competence at it. Even with the models back then, I could see how LLMs would eventually eat the act of writing software. I was beginning to understand how the Luddites felt. Livelihood, competence, identity all endangered. In old posts and videos of mine, you’ll see me torn between the melancholy of letting go of that identity, and the excitement of the fuzzy uncertain future being shaped by these new tools.

Software engineering was always a field where change was constant, and learning to keep up with new tools, languages and techniques was a core part of the job. We went from programming for individual computers, to networked computers, to mobile devices. We learned and adapted and stayed competent. But all that change had a human pace to it. Each of these waves took the better part of a decade or more to play out and for the job market to shift accordingly. For example: one could build a long, stable career as a Java programmer in an uninterrupted line since the early 2000s, keeping up with the trickle of new language and library features, and major language updates every few years.

With AI, that human pace of change is a thing of the past. If your expectations for model and harness capabilities are anchored to 4 months ago, you’re already out of date. It feels like there is never any stable ground to stand up. Like the Red Queen said, “it takes all the running you can do, to keep in the same place; if you want to get somewhere else, you must run at least twice as fast as that!” There are no best practices, no university courses, no anchors. You need to stay terminally online or follow a few key folks (Simon Willison, Geoff Huntley, Steve Yegge, among others) to understand how best to wield AI.

I acknowledge this– the uncertainty, the impossible pace, the threat to livelihood– is ultimately stressful, and often demoralizing and dejecting. All the life scripts and paths we were given are being torn up, and the new ones are not in place yet.

At the same time, the bitter truth in my opinion is that there is no return. I’ve been hiring software engineers, both at big-tech and startups, for a decade. Today, if I interviewed a candidate for a SWE role and they told me they were skeptical of using AI tools for programming, that would be the same level of red flag as someone in 2020 saying they don’t like IDEs and write their code in notepad. Is it possible they are a great software engineer? Yes, I’d keep an open mind. But it would still be a giant red flag. Also round the corner for eng managers lies the issue of handling perf with the increasing divergence between AI-maxing-SWEs and AI-skeptic-SWEs.

I’m writing all this to acknowledge that behind all the excitement, promise and hype of AI ushering in a golden new age, the on-the-ground reality of a lot of software engineers is difficult right now. What helped me through it was learning and embracing these tools and getting them to do my bidding. I realized that my deeper skills as an experienced software engineer – specification, design, discernment, taste, judgment, review, systems level thinking, architecture – are exactly what enables me to wield them as amplifiers.

Let me explain what’s happening and where I think this is headed.

The Three Abstractions (And Why They Exist)

Claude Code currently has three different ways to extend its capabilities. Each came about at different times, solving different problems.

Slash Commands are the simplest. You have a prompt you use often. Instead of typing it every time, you save it to a file and invoke it with /command-name. That’s it. A saved prompt with a shortcut.

Skills evolved from slash commands. The key difference: a skill isn’t just a markdown file—it’s a folder. That folder contains your prompt (skill.md) plus supporting resources: code snippets, HTML templates, whatever the skill needs. This lets you build capabilities that go beyond text, like parsing PDFs or generating Word documents.

Subagents do two things. First, they let you define a persona: “You are an expert code reviewer. Here’s how you do reviews.” Second—and this is the important part—they run in their own context window. They don’t fill up your main conversation.

Here’s how invocation used to work:

• Slash commands: Explicit. You type /command and it runs.
• Skills: Auto-discovered. The model sees your intent, checks its skills folder, and pulls in relevant ones. Only the name and description load initially—the full skill loads on-demand.
• Subagents: Run in their own context window, spawned when needed.

But reading the above you can probably already see the cracks appearing. Even though each abstraction had a good reason to exist at the time it was introduced, now that you have all three it makes choosing one not entirely straightforward. If you want to give the model a new capability there is a fair bit of head-scratching involved. Which abstraction best fits what I want to do? They all overlap with each other to some extent.

What Changed

The recent releases blur all these lines.

Skills can be invoked as slash commands. Skills now appear in the / menu. You can explicitly invoke them. They still auto-discover, but explicit invocation is now an option. The physical difference in how skills and commands were invoked is now gone.

Skills can run in their own context. A new front matter field, context: fork, lets a skill run as a subagent—in its own context window. This was subagents’ whole reason for existing.

Skills can declare agent dependencies. The agent field in skill front matter lets you specify which agents (explore, plan, etc.) a skill can use. The semantics here are honestly confusing. I can’t wrap my ahead around what it means for a skill to use a generic agent like “plan”.

These changes take three abstractions that already overlapped and brings them much closer together. I’d be surprised if all three didn’t collapse into one neat, easy-to-understand abstraction soon.

A Unified Abstraction

What could that unified abstraction look like?

These three abstractions conflate two orthogonal concerns:

Concern 1: How to encode knowledge.

There are two types of knowledge:

• Conceptual knowledge: Abstract domain expertise. “You are a security engineer. Here’s what great security engineers look for.” This is persona-style knowledge—the stuff you’d put in a subagent.

• Procedural knowledge: Step-by-step processes. SOPs. Code snippets that manipulate PDFs. Scripts that generate documents. This is what skills are good at.

Concern 2: Where to execute.

Do you run in the main context window, or fork into your own? This used to be a subagent-only feature. Now skills can do it too.

When you mix orthogonal concerns into the same abstractions, you get the muddy situation we have now.

Where This Is Headed

The folks at Anthropic are probably already thinking along these lines. The changelog hints suggest they are. Here’s what I think the endgame looks like:

One abstraction: Skills.

Skills are already a superset of the other two:

• They can contain just a markdown prompt (like a command)
• They can contain resources (their original purpose)
• They can encode personas (conceptual knowledge)
• They can run in their own context (like subagents)

One switch: Context mode.

A simple front matter field: context: main or context: fork. That’s it. You decide whether your skill runs in the main conversation or gets its own context window.

Composition through references.

Skills can reference other skills. This is already possible and works naturally with auto-discovery. A skill referencing another is just an emphasized hint—“when running this, keep these other skills in mind.”

Invocation becomes uniform.

Everything is /skill-name. Auto-discovery still works for capabilities the model decides to pull in. No mental overhead about “is this a command or a skill?”

The Simplified Model

That’s it. One primitive. One switch for context behavior. Composition through references. Uniform invocation.

Everything else—the distinction between commands and skills, the separate concept of subagents—becomes unnecessary.

Why This Matters

If you’re building workflows with Claude Code today, you’re navigating three overlapping abstractions with unclear boundaries. Do you make this a command or a skill? Should it be a subagent? The decision tree is more confusing than it needs to be.

A unified model means one question: “What knowledge am I encoding?” Write it as a skill. If it needs its own context, flip the switch in YAML frontmatter.

I hope we see this simplification in the next few releases. The changes in 2.1.x are clearly headed this direction.

When that happens, it’s worth understanding the mechanics of it. With that in mind, I went through building a very simple app and getting it to run and render inside ChatGPT:

Some observations:

These are very early days, in every sense: specs, tooling, patterns, and monetization. It really does feel similar to the early days of the iOS App Store. It isn’t clear yet what kind of apps will feel at home on this new platform.

Specs. OpenAI has simply made up its own conventions on top of the Model Context Protocol (MCP). Meanwhile, Anthropic has a proposed extension to MCP that is currently in the early stages of community feedback.

Tooling. There isn’t any. Other than whatever exists to write a Python or TypeScript server, or more specifically, an MCP server. Developers need to ensure they’re conforming to the OpenAI-specific conventions that the MCP server must obey (e.g., serving ui://... resources) that will make it a valid app in ChatGPT.

There isn’t even good (or any) tooling to test and iterate on your app server. Anthropic’s MCP Inspector will let you introspect and execute a generic MCP server. MCPJam claims to have an MCP inspector with support for ChatGPT apps, but I couldn’t get it working. I had to vibe-code a simple “emulator” to render ChatGPT apps locally. OpenAI’s recommended process is to turn on “developer mode” on your real ChatGPT account (which is possible only on paid plans!) and tunnel your local MCP server into it. I’ll be polite and say that is… not ideal.

Monetization. No official word yet from OpenAI, other than indirect monetization via distribution or transactions due to traffic coming from ChatGPT.

But in spite of all these shortcomings, I’m betting developers will flood the gates. One cannot just pass up an opportunity to reach an audience of 800 million weekly active users. This is right out of the aggregation theory playbook.

From the point of view of apps that have established brands, there is a deep strategic tension here. LLMs are driven by intent expressed in natural language. This leads to decoupling intent from brand name recognition. For example, on my phone, I translate the intent “I want to check out houses for sale” into finding and tapping the icon for the Redfin app. In ChatGPT, that intent will lead to the LLM selecting an appropriate app which can satisfy that intent. It could be Redfin or Zillow or any number of other real estate apps. I as a user don’t need to know or care. I’m trusting ChatGPT to simply pick the right one and the best one. I riffed on this a bit more here:

There are many unknowns. As Ilya Sutskever said: the answer will reveal itself.

It’s no accident that the most successful AI agents today–CLI agents like Claude Code–are built to run in a Unix environment. The fit is so natural, so seamless, that it’s easy to miss how remarkable it is. But understanding why Unix and AI agents work so well together can teach us a lot about how to build robust, reliable agents.

It starts, first of all, with the Unix Philosophy: small, sharp tools that do one thing well; orthogonality and the primacy of composability; and text as a universal interface.

And while Unix tools are sharp and focused, they also have depth and richness within their domain. For example, wc can count words, lines, characters or bytes. I call this the “narrow doors, large rooms” principle. This ensures that a tool can cover all or a large part of a given problem area.

If you’ve struggled with building LLM tool-calling systems, you will find that these are exactly the ideas that also help to untangle them and build them more reliably (1, 2).

You can invert some of these design principles to see common pitfalls in LLM tool-calling: “wide door, shallow room” (a vaguely-named tool that doesn’t do much) and “wide door, labyrinth inside” (a tool that does everything but unpredictably).

In addition to this core design philosophy, the early culture around Unix had a strong emphasis on writing. They wrote not just comprehensive technical documentation describing how to operate the system (man pages), but also articles and books on system design, architectural tradeoffs, and how to build complex systems from simple parts. They had an obsession with wordsmithing.

Decades later, the output of this extensive culture of writing became part of the corpus that trained modern LLMs. This is why a CLI agent feels right at home in the Unix shell. It understands all those sharp little tools, and how to string them together to achieve complex ends.

An LLM is, in a profound sense, exactly the user Unix was designed for: an entity that thinks in natural language, can read documentation, can reason about it, plan with it, and compose small operations into larger workflows. In human form, this would be a knowledgeable poweruser.

This makes for an interesting contrast with GUIs, which tried to abstract away this depth and knowledge required for effective use of a system in the name of user-friendliness. To be useful in a GUI environment, an agent would need computer vision, cursor control, window management, and pixel-level understanding of interface elements. I’m sure computer-use models will get better, but they’re miles behind CLI agents currently.

Unix was a love letter to agents. It just took fifty years for the recipients to arrive.

But in this piece I want to convince you that coding agents are actually outliers when it comes to designing and building AI agents. The very characteristics that make coding a perfect domain for AI agents—deterministic outputs, clear success and failure states, rich tooling, immediate feedback loops—are precisely what makes them unrepresentative of most other agent domains.

The success of coding agents has created a template that everyone seems eager to copy. But this template is misleading. What works for coding agents doesn’t necessarily work elsewhere, and understanding why is crucial for anyone trying to build AI agents in other domains.

Building your own tools

The people who build AI models are themselves programmers and computer scientists. Programming is how they build their products. This creates a unique dynamic that doesn’t exist in any other domain.

When researchers at the major model providers improve their models’ coding abilities, they immediately become the first line of users. The same (or very adjacent) teams build the agentic scaffolds around these models. They use these tools to write the very code that improves the models. They experience every frustration, every limitation, and every breakthrough firsthand. This isn’t just dogfooding—it’s a tight feedback loop where the builders’ own productivity depends on the quality of what they’re building. This feedback loop goes straight back into model development. No other domain has this level of builder-user overlap, and it shows in the results.

Beyond this human feedback loop, there’s the matter of training data. Programming is very well represented in the datasets used to train large language models. Code repositories, technical documentation, tutorials, blog posts—the internet is awash with programming content.

But it’s not just the code itself. The entire ecosystem of software development is documented online: testing practices, CI/CD pipelines, debugging strategies, architectural patterns, code reviews, post-mortems. This world knowledge makes coding agents not just good at writing code—they understand the full software development lifecycle because all of this knowledge is in their training data. They know that after writing code, you should execute tests. They know what test-driven development means. They know how to set up deployment pipelines. They understand version control. This depth of representation is unique to programming.

Compare this to other highly specialized technical domains—material science, drug discovery, energy, manufacturing etc. Unlike programming, there is very little public world knowledge explaining and documenting the day-to-day work tackled by a professional in such fields.

Agents Versus Workflows

Slight tangent: we need to clarify the distinction between workflows and agents.

Workflows are predetermined sequences of steps designed to accomplish specific goals. When you use a workflow, you know exactly what steps will be taken and in what order. You might use AI or LLMs to help execute individual steps, but the sequence itself is fixed. The model isn’t deciding what to do next; it’s just helping you do it better. Workflows are perfect for repetitive, well-defined tasks where consistency and predictability matter more than adaptability.

Agents, on the other hand, perform reasoning and planning. They look at a problem, consider various approaches, and generate their own sequences of steps to solve it. They adapt their approach based on intermediate results. When something doesn’t work, they try something else. They might ask for clarification or suggest alternative strategies. True agents don’t just execute; they plan and adapt.

Coding agents are firmly on the agentic side of this spectrum. In fact, most coding agents have a special affordance for “plan mode”. The extensive programming-related world knowledge baked into the models lets them look at every unique programming problem they are given and improvise a high-level plan to tackle it.

However, most domains that people are trying to apply agent architectures to actually just need better workflows. Not every problem benefits from dynamic planning and reasoning. Sometimes, what you need is a well-designed sequence of steps executed consistently. (What precisely is the role of an LLM in such a workflow? That is a topic for another post.)

The Tacit Knowledge Problem

The real reason coding agents are outliers becomes clear when you look at other specialized domains. In fields outside programming, most of the knowledge needed to operate effectively isn’t written down anywhere, public or not. It’s tacit knowledge, locked inside the heads of experts and professionals. They notice subtle patterns, remember similar cases, and apply intuition. This knowledge is contextual, situational, and often very hard to fully articulate.

Building agents for these domains requires solving this tacit knowledge problem. You need to somehow extract knowledge that experts themselves often can’t articulate, codify it in a way that preserves its contextual nuance, and then ground an AI model on it. This is fundamentally harder than leveraging the already-documented world of programming knowledge.

Codifying this tacit knowledge and workflows, and providing it to a model to ground on is exactly the core playbook for the “AI Agent for X” startups (X could be lawyers, accountants etc). This is not easy. You either need to be a domain expert, or else embed yourself with them for an extended period of time. There are no shortcuts. (Some ways to do that is yet another topic for another post)

Why Success Doesn’t Translate

The success of coding agents has created unrealistic expectations for AI agents in other domains. We see these impressive coding demonstrations—an agent building an entire web app from a prompt, fixing complex bugs, refactoring legacy code—and assume similar breakthroughs are just around the corner for other fields.

But the conditions that enable coding agents are unique, not universal. Programming has deterministic outcomes: code either works or it doesn’t. It has immediate feedback: run it and its tests and see what happens. It has rich tooling: debuggers, profilers, test suites. It has clear success metrics: tests pass, performance improves, bugs are fixed. And most importantly, the people building the models are programmers and use the models for programming.

Most other domains lack these characteristics. Success in marketing is measured in customer engagement months later. Legal advice quality might only become clear after a case concludes years hence. Medical treatment effectiveness varies by patient in ways that are hard to predict. Design quality is subjective and context-dependent.

This doesn’t mean we can’t build useful AI agents for these domains. But it does mean we need to adjust our expectations and our approaches. The coding agent template is something to aspire towards, but not a given in other domains.

I like working with developers who communicate. Ask any of them. I stress this all the time. “Narrate your work,” says Dave. Our RFC process generates excellent writing. Perhaps we have different values than other devteams. People wonder how we get so much done with so few people. Perhaps this is it.

Jon Udell echoed the sentiment a few years later (2009):

Since then I’ve spoken a few times about the idea that by narrating our work, we can perhaps restore some of what was lost when factories and then offices made work opaque and not easily observable. Software developers are in the vanguard of this reintegration, because our work processes… are fully mediated by digital networks. But it can happen in other lines of work too, and I’m sure it will.

An important part of early Google engineering culture was the notion of snippets. Snippets were short summaries of work and achievements from the last week, and sometimes, plans for the next. Most engineers would write snippets for their week every Friday afternoon.

Importantly, snippets were visible to the entire company. Anyone could look up anyone else’s snippets and hence read an ongoing log of their work, their challenges and their plans.

They ranged in length and format. Most wrote snippets that were a quick list of bullets. Often there would be links to documents, bugs or code. But there were a few outliers who used snippets as a creative outlet, writing mini-essays or sometimes long reflections in their snippets. I greatly enjoyed these.

Narrating your work is different from making your work public. It is the meta-narrative. It is talking about how you did the work, about the thought process behind it, about what informed the work. It is about all the missteps and the struggle and the paths that did not work out. It is not a clean story. It is not a synthesized result. It is a lab notebook, not a published paper.

Ironically, when you narrate your work, there usually isn’t a narrative. Every individual update seems to be a random point in the cloud. Only when you look back on a few months or years of you narrating your work do you begin to see all the threads. All those points start appearing as a coherent picture when you zoom out.

Narration serves both the writer and the reader. To the writer it provides a respite, and recognition of work accomplished, lessons learned and things built, in an otherwise breakneck work environment. To the reader it provides commiseration, comfort in the knowledge that there are others sharing similar burdens and pushing through comparable struggles. Above all—the reader gets to learn from those further along the path of mastery.

Keep narrating.

The Atom of Work: The Read-Cognify-Write Loop

Break down any task performed by a knowledge worker, and you find the same atomic structure repeating itself:

• Read: You ingest information. This could be reading an email, listening in a meeting, analyzing a spreadsheet, or even reviewing the output of your previous task.
• Cognify: You process that information. This is the black box of thought where judgment, synthesis, analysis, extrapolation, or creativity happens. It’s the process of forming a judgment, decision, or the next step.
• Write: You persist that new insight. This could mean writing an email, updating a document, creating a presentation slide, or writing a piece of code.

Complex projects are a long series of these read-cognify-write steps. A massive undertaking like launching a new product is composed of thousands of smaller RCW loops. Sometimes they’re strung together in bursts carried out in a few minutes, other times they might have weeks-long pauses.

The RCW steps need a structure to live within. First, you must create a project plan. To do that, you break the task down: “analyze market data” (a series of RCW loops), “draft a budget” (more loops), “assign team roles” (even more loops). At the end, you perform a final synthesis step, combining the outputs of all these threads into the final artifact—the project plan.

That sounds like the basic plan-and-execute structure of most AI agents today, and it is.

For decades, humans have been the sole engine capable of this entire process. We hold the high-level intent for a project over weeks, months, or even years, manually “compiling” that intent into countless atomic RCW tasks. We are the planners, the executors, and the synthesizers. But this is changing.

But agents have been progressing to hold intent over longer and longer tasks. The resulting curve showing the length of tasks that AI can do shows up as a neat line on a log-chart, i.e. a clean exponential.

The AI “Compiler” for Knowledge Work

The latest generation of reasoning AI models can now perform both the low-level execution and the high-level planning and orchestration. They can not only perform an individual RCW step but also do the planning required to decompose a complex goal into those smaller steps and then synthesize the final output.

AI is becoming like a compiler for knowledge work. Tiny RCW loops are the target assembly language ISA for these compilers.

When a programmer writes in a high-level language like Python, they are expressing their intent. They don’t worry about memory allocation, CPU instruction sets, or register management. The compiler handles that, translating the simple, human-readable intent into efficient, low-level machine code. Yet, there was a time when humans had to painstakingly handle moving values between memory and registers, manage the stack, and write assembly language.

Similarly, we will soon stop managing the low-level “assembly language” of work. Instead of manually executing hundreds of RCW loops, we will state our high-level intent—“Create a go-to-market strategy for Project X”—and the AI agent will compile it, orchestrating the necessary steps to produce the final output.

The Bitter Lesson and the Black Box

Ethan Mollick makes a similar point eschewing this low level work in favor of clearly stating intent, making the analogy with Sutton’s “The Bitter Lesson.” The lesson is that we shouldn’t try to force AI to replicate the exact steps a human would take. Instead, we should give it a clear goal and a robust way to measure success, then let it find its own, potentially alien, path to the solution.

This is exactly how we treat compilers. A programmer rarely, if ever, inspects the assembly code their compiler generates. They trust it because it’s proven to be efficient and correct. We must adopt a similar mindset for AI agents. The key is not to micromanage their internal plan but to focus on two things:

• Clearly Stated Intent: The quality of your output depends entirely on the clarity of your prompt. Our skill will shift from execution to direction.
• Inspectable Artifacts & Evals: We don’t need to know how the AI did it, but we need to trust the result. This means demanding outputs with citations, verifiable reasoning, and clear diffs showing changes.

Optimizations for Knowledge Work

Pushing the compiler analogy further, you could “optimize” knowledge work very much like how a compiler optimizes code when emitting assembly. AI agents will apply the same principles to knowledge work workflows.

Consider these classic compiler optimizations and their application to corporate knowledge work:

• Dead Code Elimination: A human, working from habit, might waste a day generating a report that no one uses in the final presentation. An AI agent, seeing the full dependency graph of the project, would identify the task as “dead code”—its output is never used—and eliminate it before it even begins.
• Common Subexpression Elimination: A human might analyze a dataset for a summary email, then someone else might re-analyze the same raw data for a detailed slide deck. An agent would perform the core analysis once, cache the results (the “common subexpressions”), and then reuse them.
• Instruction Reordering & Parallelization: A human preparing a competitive analysis would research funding, then product features, then press releases, one after another. An agent would see these as independent tasks and execute them in parallel, cutting the time-to-completion dramatically.
• Loop Unrolling: A human sending 50 personalized follow-up emails performs a slow, repetitive loop with high cognitive overhead for each iteration. An agent would “unroll the loop,” pulling all the data in one batch operation and generating all 50 drafts simultaneously.

From Assembly to Intent

The knowledge work we do today is a mesh of high-level strategic thought and low-level, manual execution. We are the thinkers and the typists, the strategists and the assemblers. The AI compiler is coming for the assembly part of the job.

Our value and focus will shift entirely to the realm of intent. Our core tasks will be to ask the right questions, to define the goals with clarity and precision, and to build the robust evaluation systems that ensure the quality and reliability of the output. The painstaking, manual process of bringing those ideas to life will be abstracted away, just like compilers abstracted away assembly language to high-level languages.

No-hands editing, processing and publishing

The goal of this video production stack is to take a raw video, then automatically edit it, transcribe it, come up with timestamped chapter markers for the YouTube description, and upload it to my YouTube channel. Basically this is a fully automatic publishing workflow after the point I hit stop on the record button.

This, of course, is very specific to the kinds of videos that I make, which is talking head videos with a screen share. But that is a large enough genre on YouTube that this setup might be useful to others out there.

I have several different tools across multiple git repos to accomplish this. Today I am making all those repos public. Let me explain below what each one does.

The Complete Workflow

My YouTube publishing workflow consists of four main components, each handling a different part of the video processing pipeline. They’re written in Python and designed to work seamlessly together:

1. color_edit - Intelligent video editing based on visual cues
2. yt_chapter_maker - AI-powered chapter generation and title suggestions
3. yt_upload - YouTube API integration for uploading
4. video_upload_workflow - The orchestrator that ties everything together

Let me walk you through each component and how they work internally.

color_edit: Visual Cue-Based Video Editing

The first and most crucial tool in my stack is color_edit. This is the tool that automatically creates a fully edited video from my raw recording. Before I created this tool for myself, I used to spend multiple hours reading each video.

The key innovation here is using colored frames as editing markers. When recording, I display:

• Green frames to mark sections I want to keep
• Red frames to mark sections I want to remove
• Regular content is neither red nor green

This is much easier to demonstrate than to explain. Check out this video where I visually explain how this works.

Another important function performed by this tool is the removal of silent parts of the video, also known as “dead air”.

The final output is a tightly edited video with no dead air and only the sections I explicitly marked as keepers.

yt_chapter_maker: AI-Powered Chapter Generation

Once the video is edited, yt_chapter_maker takes over to generate metadata. This tool is pretty simple. It prompts an LLM to look at the transcript for the video and then suggest few options for titles as well as timestamped chapter markers that can go in the video description.

yt_upload: YouTube API Integration

The yt_upload tool handles the actual YouTube upload process using the YouTube API. You will need to get client secrets using the Google Cloud console in order to invoke the API. Also, the very first time you run it, you will have to do an OAuth dance through the browser.

The uploaded video is kept as private so that you look at the video and its metadata before publishing it.

video_upload_workflow: The Orchestrator

Of course I don’t want to have to invoke all of the above steps individually every time. So I have a meta-orchestrator that runs the entire pipeline. It also takes care of transcribing the video using OpenAI’s whisper tool.

There’s some caching of intermediate outputs to make retries faster in case something breaks midway.

Audio

Getting good audio is way more important than high-resolution video. People can watch a potato webcam video if it has decent audio, but will not stay for 2 seconds on video shot on an 8K RED camera if the audio sounds like it is coming out of a tincan.

Fortunately, it is very hard to go wrong with any decent modern microphone. These are the mics I’m currently using (depending on where I am recording).

Experience

Various parts of the above setup came about organically over time.

color_edit is the oldest one. I’ve been using it for more than 4 years now and every video on my channel in that time frame has been automatically edited using it. I haven’t even been making any changes to it recently. It does its job well and without any fuss.

The other parts are more recent and were mostly cobbled together in the last year or so. All the same, at this point they have been used to publish many videos on my channel and I intend to keep using this pipeline.

This might seem like a controversial or nonsensical statement, but consider the following:

In Claude Code, you can define custom commands and workflows in plain text:

Custom slash commands allow you to define frequently-used prompts as Markdown files that Claude Code can execute.

(Gemini-CLI has a similar feature, with the slight syntactic difference that it uses TOML.)

The initial usage of these commands started by automating common developer workflows (write, test, review, commit, push, bring up dev server etc). They were “scripts”, but in natural language.

In a shell or Python script one could never say “look at the diff and formulate a succinct but descriptive commit message following our company style guide at <url>”. In current coding agents, such a semantic task is trivial.

But then, soon after, people started using these agentic coding agents as everything agents. See the long list of non-coding use-cases in this X thread. Or how one can run Claude Code in the home directory (not a git repo) and use it to automate most of what one does on a computer.

This uplevel of using CLI agents as everything-agents brings the same conundrum that every developer faces when faced with a series of repetitive tasks: should I bite the bullet and write a script to automate this, or just go ahead and do it manually? Almost always, writing a script takes much more time than doing in manually once. Now instead of writing a shell or Python script, one has to do the calculus on slowing down and writing out a text file with instructions and criteria, so that the agent can do it in the future.

Stuck in Chatland

While developers are automating entire workflows with coding agents, the rest of us are stuck in what I call “chatland”. Chatland is using ChatGPT to “ask questions”, or run prompts one-shot. And then what? Closely read it? Copy/paste it somewhere? Ask even more questions?

Chat is great for exploration; it is not meant for repeatability and workflow automation. It is a nice local maxima, certainly not a global one. One could, however, use a long conversation with the model to arrive at the process and pull out tacit knowledge, much like a newbie on the job does with a more experienced colleague.

A whole host of tools have cropped up to address the shortcomings of chat for automation: n8n, Gumloop, make.com and many others. Their drag-and-drop UI based on flowchart-construction seems like a natural way to express and then automate workflows, but it can be fiddly and breaks down with data-intensive flows. As Erik Meijer put it:

Most workflow tools, like UIPath, Workato, and Azure Logic Apps, primarily use sequential composition with limited control flow and repetition validating our design choices. One complication in workflow tools is the need for a “language” of functions to manipulate data as it flows between actions. We replace this with pattern-matching to simplify and enhance the expressiveness of the base language without the need for an extra layer of ad-hoc functions

(I covered that paper in a video here.)

More importantly, when businesses do document their workflows, they rarely use flowcharts (or the more enterprise-y BPMN). They write SOPs (standard operating procedures) in natural language. SOPs are the grown-up versions of custom slash-commands written in markdown.

The pain of articulation

In a recent post titled “AI is a mirror”, I wrote:

The company that has a clear, pre-defined workflow that is even ready to be considered for agentification is extremely rare. They know they have inefficiencies. They feel the pain of manual, repetitive tasks. But they can’t articulate the precise sequence of steps, the decision points, and the logic that governs the work.

Ethan Mollick had already picked up on this theme. In a recent column, he spoke of the “garbage can” model of the organization and its processes:

One thing you learn studying (or working in) organizations is that they are all actually a bit of a mess. In fact, one classic organizational theory is actually called the Garbage Can Model. This views organizations as chaotic “garbage cans” where problems, solutions, and decision-makers are dumped in together, and decisions often happen when these elements collide randomly, rather than through a fully rational process… The Garbage Can represents a world where unwritten rules, bespoke knowledge, and complex and undocumented processes are critical. It is this situation that makes AI adoption in organizations difficult, because even though 43% of American workers have used AI at work, they are mostly doing it in informal ways, solving their own work problems. Scaling AI across the enterprise is hard because traditional automation requires clear rules and defined processes; the very things Garbage Can organizations lack.

He goes on to apply the bitter-lesson mindset to applying AI to enterprise work:

The Bitter Lesson suggests we might soon ignore how companies produce outputs and focus only on the outputs themselves. Define what a good sales report or customer interaction looks like, then train AI to produce it. The AI will find its own paths through the organizational chaos; paths that might be more efficient, if more opaque, than the semi-official routes humans evolved.

(Emphasis mine.)

I agree, and this is why I think we must encode knowledge of “what good looks like” for the output of a process into a rigorous eval/benchmark.

The client company defines the task by creating this benchmark. It’s filled with input examples and the corresponding ideal outputs. The agent-building company’s only job is to build an agent that achieves a pre-agreed-upon score on this benchmark.

Note that raw model intelligence will not help with workflow articulation. As Casper put it:

Recently, both OpenAI and Google models were on par with gold medallists in the International Mathematical Olympiad 2025 (IMO). At the same time it’s still difficult to make AI agents work for relatively simple enterprise use cases. Why is there such a disparity in model performance between problem domains? Why are models so much better at complex maths tasks that only few humans can complete, while struggling at simple every day tasks done by most humans? It’s because the bottleneck isn’t in intelligence, but in human tasks: specifying intent and context engineering.

(Emphasis mine.)

Developers writing elaborate workflows in plain-text files are speed-running the same journey that enterprises need to make to achieve the end-to-end benefits of AI. They have to look beyond helping individual contributors with granular tasks in “chatland AI”, and make the leap to “workflow-automation AI”.

Recently I became Claude Code-pilled. I had been eagerly trying out the CLI agents from all three major providers: Gemini CLI from Google, Codex CLI from OpenAI, and Claude Code from Anthropic. All the breathless AI influencers turned out to be right: Claude Code is just much, much better than the others.

If you look at the internal architecture of these CLI agents and the resulting full product, you will see the Mac vs PC story playing out all over again.

The architecture of a CLI agent consists of three layers:

1. The Model: This is the foundational Large Language Model (LLM) that does the core thinking—the Claude, Gemini etc. This layer is responsible for the raw intelligence of the system.
2. The Agentic Scaffolding Around the Model: This middle layer consists of all the tools, prompts, retrieval mechanisms (like RAG or agentic search), and retry loops. It’s the framework that guides the model, giving it the tools and context it needs to perform a specific task, like fixing a bug, or writing code for a feature spec.
3. The Text User Interface (TUI): The textual user interaction in the terminal. This layer manages everything the user sees and interacts with: rendering output in the terminal, handling input, and displaying the status of ongoing tasks.

In most of the major CLI agents today—Codex, Gemini CLI, and Claude Code—these three layers are bundled together into a single, monolithic package. You can’t easily swap out one component for another.

But just like people wanted to build their own PCs out of parts of their choice, we’re now seeing CLI agents that are offering a more mix and match approach. Agents like OpenCode and Crush let you swap in any LLM and also bring modularity to the core system prompts used for agentic behavior. They are innovating at the TUI level and pulling in users with eye-candy UI. They are betting on a modular, mix-and-match strategy.

Anthropic, however, is taking the complete opposite approach with Claude Code. Their strategy is reminiscent of Apple’s: a fully integrated, hermetically-sealed system. You use the Claude model with their proprietary agentic harness and their TUI. It is telling that of the three major CLI agents, it is the only one that is not open-source.

It works though. Claude Code has cemented its reputation as the best SWE (and even general-purpose) CLI agent. I suspect a lot of that is due to the underlying Claude models, but also a significant part of its effectiveness comes from the tenacity and intelligence baked into its agentic harness—the secret sauce they’re not giving away (prompt reverse-engineering notwithstanding).

I’m not sure which approach will win out. Maybe we’ll settle in an equilibrium–very much like the Mac and the PC have–where both kinds of products are successful and have a large user base.

I’ve talked to a lot of companies about building AI agents, and I’ve noticed a strange pattern. You sit down with them, ready to talk workflows, pipelines, and agents, and you ask the simple question: “What do you want to automate?”

And they don’t know.

AI acts as a mirror. You bring it in, hoping it will solve your problems, but it just stands there and reflects your own organization back at you, asking: “So, what do you actually do here? What is your process? What is your workflow?”

For a shocking number of companies, the answer is a shrug.

The company that has a clear, pre-defined workflow that is even ready to be considered for agentification is extremely rare. They know they have inefficiencies. They feel the pain of manual, repetitive tasks. But they can’t articulate the precise sequence of steps, the decision points, and the logic that governs the work.

The initial enthusiasm for “AI transformation” quickly becomes a painful, soul-searching exercise in process documentation. The AI doesn’t tell them what to do; it forces them to figure it out for themselves.

The Second Battle: Escaping the “Vibes” Trap

Let’s say you get past that first hurdle. After weeks of dialogue, whiteboarding, and untangling departmental spaghetti, you finally pin down a workflow. You build an agent. It starts doing the thing.

Now comes the second battle: Is it any good?

The default metric for success in most organizations is “vibes.” Does it feel right? Does the output seem okay? This is a terrible way to build an engineered system, which is what an AI agent is. “Good enough” isn’t a spec.

This is where the mirror shows its second reflection. Not only do you have to define the process, but you also have to define what a successful outcome looks like. You need a rubric. You need properties of a good output. You need a benchmark.

Without a clear definition of “good,” you’re just tweaking prompts and hoping for the best. You can’t measure progress. You can’t have a meaningful conversation about improving the agent’s performance. You’re stuck in a subjective loop of feedback based on gut feelings.

To truly make the most of AI, organizations first have to look within.

Imagine two parties: the company that needs an agent and the company that builds it. In this ideal world, they never have a meeting. They never speak. Their entire interaction is mediated through a single, shared Git repository.

This repository contains one thing: a comprehensive set of evaluation data—a benchmark.

The client company defines the task by creating this benchmark. It’s filled with input examples and the corresponding ideal outputs. The agent-building company’s only job is to build an agent that achieves a pre-agreed-upon score on this benchmark.

If the client wants to change the agent’s behavior? They don’t send an email or schedule a call. They add new data points to the benchmark that illustrate the desired change. Or they modify existing ones. The agent’s behavior is shaped only by the data in the eval set.

The builder’s job is simple: make the agent pass the test. The client’s job is also simple: create a test that accurately reflects their needs.

If this sounds familiar, it should. It’s essentially Test-Driven Development (TDD) for AI. In software engineering, TDD dictates that before you write a single line of feature code, you first write a test that defines what success looks like. The test fails, naturally. Then, you write just enough code to make it pass. Plato’s Cave of Evals applies the same philosophy. The benchmark is the failing test. The agent is the code you write to make the test pass.

Of course, this is a hypothetical ideal. In the real world, the nuance of a business problem can’t always be perfectly captured in a static dataset. But Plato’s Cave of Evals captures the spirit of what we should be aiming for. It forces clarity. It makes the abstract concrete. It shifts the entire development process from subjective feedback loops based on vibes to objective, data-driven iteration.

This isn’t an application-layer problem. This is an identity-layer problem.

The Core Problem: Impersonation vs. Delegation

Right now, an AI agent has two bad options for accessing your accounts:

1. Scoped API Access: The agent gets an OAuth token with a limited scope (e.g., calendar.read). This is a sandbox, and a pretty small one at that. The agent can’t browse the web as you; it can only hit pre-approved endpoints. This neuters the whole concept of a general-purpose agent.
2. Credential Handover: You give the agent your literal username and password. This is catastrophically insecure. It makes auditing impossible and turns every agent into a potential supply chain attack vector for your personal accounts. Every action the agent takes is logged as an action you took.

We have robust protocols for a user authenticating themselves, and for an application acting on its own behalf. We have no standard for an application acting as a delegate for a user, especially for today’s use-cases of AI agents needing to work as the signed-in user.

No, Service Accounts Are Not the Answer

Let’s get this out of the way: service-accounts are the wrong abstraction. A service account is an impersonal, non-human principal. It’s for a CI/CD pipeline or a backend cron job.

A personal AI agent is the opposite: it’s a delegate for a specific human, operating on that human’s personal data. Tying its actions to billing-reporter@gserviceaccount.com is useless. We need to tie its actions to jane.doe@gmail.com but with a clear, auditable asterisk.

The Fix: Delegated Auth with an actor Claim

The fix has to happen at the identity layer to be universal and widely applicable. The major identity providers—Google, Microsoft etc—need to extend their auth protocols (think OAuth/OIDC) to support agent delegation.

When a user authorizes an agent, the IdP should issue a standard access token with one crucial addition: a claim that specifies the actor. Something like:

“actor”: { “type”: “delegated_agent”, “id”: “agent_id_xyz” }

When an agent presents this token to a resource server (like the Gmail API), the server can see two critical pieces of information:

1. The Subject (sub): The action is for jane.doe@gmail.com.
2. The Actor (actor): The action was performed by agent_id_xyz on Jane’s behalf.

This creates an audit trail for actions performed by an agent on behalf of a user. It also allows us to build policy around the actor. The IdP or the resource server could enforce rules like actor.type: ‘agent’ cannot perform destructive actions like account.delete or password.reset.

Why doing this at the app layer doesn’t work

Trying to solve this at the app layer is a non-starter. You’ll get a thousand different, incompatible implementations, and no one will trust anyone else’s X-Agent-ID header. The IdP is the only trustworthy source of truth for who is acting. It’s the only entity that can cryptographically certify that a given request was performed by a legitimate, user-sanctioned delegate.

Immediate use-case: Agentic Browser Support

The need for something like this is already apparent in today’s agentic browsers, like Comet, or Google’s upcoming Mariner.

The ideal implementation is native to the browser itself. Imagine Chrome’s profile manager having a concept of “delegated agent sessions.” An agentic browser like Mariner would be able to clearly distinguish actions performed by the human user from those performed by the browser agent carrying out its own plans and tasks.

The benefits are clear:

• Users: Get to delegate tasks without handing over the keys to the kingdom. You get a real audit log and systemic guardrails.
• Developers: Can finally build powerful agents without resorting to janky screen scraping or insecure credential storage.
• Identity Providers: Unlock a new ecosystem of trusted third-party agents, making their platforms stickier and more powerful.

Without a proper identity layer for delegation, personal AI agents will remain stuck between being glorified toys and unacceptable security risks. It’s time to build the auth protocols for the agentic era.

The New York Times recently published an article about programmers at Amazon complaining that their jobs were beginning to look like warehouse work because of AI.

At Amazon, Some Coders Say Their Jobs Have Begun to Resemble Warehouse Work Pushed to use artificial intelligence, software developers at the e-commerce giant say they must work faster and have less time to think. Others welcome the shift.

Their managers are turning the screws on them, saying they should be using AI to get more done in less and less time.

Three Amazon engineers said that managers had increasingly pushed them to use A.I. in their work over the past year. The engineers said that the company had raised output goals and had become less forgiving about deadlines.

This is where the comparison with blue-collar warehouse workers comes in. These programmers are now saying that they feel like “bystanders in their own jobs”.

The automation of coding has special resonance for Amazon engineers, who have watched their blue-collar counterparts undergo a similar transition.

They’re making the same argument that factory and warehouse workers have been making for a long time: their jobs have been stripped of all creativity and craft, becoming mechanized, repetitive, and soul-crushing.

The new approach to coding at many companies has, in effect, eliminated much of the time the developer spends reflecting on his or her work. “It used to be that you had a lot of slack because you were doing a complicated project — it would maybe take a month, maybe take two months, and no one could monitor it,” Dr. Katz said. “Now, you have the whole thing monitored, and it can be done quickly.”

“White-collar" programmers are now feeling the same pressures that have been felt in mechanical factory work since the dawn of the industrial age.

I want to tie these recent developments back to some very old ideas that go back to the very birth of the field of software engineering itself—and hopefully shed light on some common themes that have been recurring in our field since the very beginning.

Blue-collar coders

Back in 2013, I wrote a blog post reacting to some prominent bloggers talking about “blue-collar coders.” They were arguing that the work of web and software development can and should be thought of as blue-collar work.

Here is Anil Dash writing back in 2012 – he is making the case for software engineering to be treated as vocational education. “We need web dev vo-tech”.

High schools have long offered vocational education, preparing graduates for practical careers by making them proficient in valuable technical skill sets which they can put to use directly in the job market right after graduation. Vocational-technical schools (vo-tech) provide trained workers in important fields such as healthcare, construction trades, and core business functions like accounting. For a significant number of my high school peers, vo-tech was the best path to a professional job that would pay well over the duration of an entire career.

Now it’s time that vo-tech programs broadly add internet and web technologies to the mix. We need web dev vo-tech.

And here is Laurie Voss making a similar argument about how knowledge workers should think of themselves more as blue-collar workers. He makes the crucial comparison to factory work and Taylorism. Voss argues for that same model to be applied to web development.

Think about how a physical factory worked. The reason unskilled jobs in manufacturing, say, cars existed is because some very highly skilled people first got together and looked at the process of building a car and said “okay, we can automate this bit and this bit and this bit, but making machines to do this bit is way too hard”. The blue collar workers of Detroit didn’t know how to build cars; they knew how to operate one particular bit of machinery. The only people who knew how to build cars were the guys who put the machines together.

Now let’s try applying that model to web development, an example I pick because I know a fair bit about building web sites. Think about all the businesses in the world that have web sites, or need them built or maintained. There is an entire industry built around cranking out simple websites for small businesses, in WordPress or Drupal or a thousand proprietary solutions. This industry is making a bunch of smart freelancers a ton of cash, which is great for them individually, but terrible for the tech sector as a whole.

This is an argument that can be applied to any technology that’s fairly mature, where the methods for achieving a certain outcome or building a certain product are well-understood. It’s not surprising that even back in 2012, when web development was already mature with a fairly well-understood stack, he would make an argument like this.

My old blog post reacting to these two was making the opposite case. I argued that it’s not tenable to think of software development purely as a vocational line of work, simply because of the pace at which technology changes.

The problem with this vocational tech approach is that it implicitly assumes the knowledge gained during vocational training will be relevant for a long period of time. The pace of technology change renders that unviable.

But the core philosophy that all these people are talking about is Taylorism. And Taylorism is also deeply linked to the birth of software engineering. In another very old blog post I reflected on Taylorism and how it relates to programming. Taylorism is a deep topic, but if you were to drastically simplify it, its core ideas are these two: you measure everything, and you separate thinking from doing.

Briefly, Taylorism has two central tenets:

1. Measurement: associate metrics with all aspects of work.
2. The separation of thinking and doing: An educated class of managers measures and plans all the work, and is responsible for the overall process, while a class of laborers carries out the implementation of those plans.

This is exactly the point Voss was making when comparing software development to what happens in a car factory. Someone thinks about, examines, and codifies a process (the thinking part), and they then bring in an army of laborers to do tiny bits of that process (the doing part).

The birth of software engineering

Where this links up to software engineering is that back in 1968, the very term “software engineering” was coined at a NATO conference of the top people building computers and writing software at the time. One of the central themes the attendees were discussing was the notion of industrializing the process of building software. Doug McIlroy expressed this by saying, “The software industry is not industrialized,” and compared it to the mass production techniques used in physical factories.

Software production today appears in the scale of industrialization somewhere below the more backward construction industries. I think its proper place is considerably higher, and would like to investigate the prospects for mass-production techniques in software.

He was calling for a catalog of interchangeable parts with well-understood characteristics, and for applying the same methods used for physical assembly in factories to the creation of software.

The idea of subassemblies carries over directly and is well exploited. The idea of interchangeable parts corresponds roughly to our term modularity, and is fitfully respected. The idea of machine tools has an analogue in assembly programs and compilers. Yet this fragile analogy is belied when we seek for analogues of other tangible symbols of mass production. There do not exist manufacturers of standard parts, much less catalogues of standard parts. One may not order parts to individual specifications of size, ruggedness, speed, capacity, precision or character set.

Another lens through which you can view this entire argument is what I call the notion of operation versus expression. The key idea is that sometimes you’re operating a piece of machinery, and the goal is to quickly achieve some well-defined end. But other times, you’re trying to express yourself, which goes back to the notion of looking at what you’re doing as a craft.

Software engineers like to talk about how they craft their software. Are you operating a piece of machinery, or are you trying to express yourself and create something? This argument comes up again and again in the software industry whenever there’s a big shift in how we build software. Even when IDEs were becoming popular, the idea that they were taking the craft away from software engineering was widely expressed. Many software engineers felt that IDEs were constraining the creative ways in which they could write software.

This argument is now, of course, coming up again because AI is once again fundamentally changing the way we write software. But the point of this whole rant is that going back to the very birth of the field of software engineering, we’ve wanted to make it industrial. We’ve wanted to make it Taylorist. And there has been a steady stream of developments—from compilers and high-level languages to IDEs and now, finally, AI—that have made us steadily march in that direction.

So, what is Jevons Paradox? It’s the phenomenon where making a resource more efficient to use doesn’t actually lead to us using less of it. Paradoxically, it often leads to increased consumption of that very resource. Because it’s more efficient and often cheaper or more accessible, more people start using it, or they find more purposes for it, and the total consumption shoots up.

Why the sudden chatter around this specific time, early 2025? A big piece of news hit around January 20, 2025: DeepSeek R1 was released. China being denied access to the most powerful NVIDIA GPUs, they had to fall back on less powerful hardware. The efficiency with which they trained that model took most AI watchers by surprise. NVIDIA’s stock took a nosedive in the couple of days following the release.

But apart from the plate tectonics of AI, there’s a very personal side to this.

Venkatesh Rao put it perfectly in a note:

Good sign that AI Jevons paradox has kicked in properly. I’m working much harder than before AI. It’s now clear that I was lazy and low productivity before because I couldn’t afford serfs/graduate student assistants.

Those “graduate student assistants” he’s referring to are the latest AI models, of course, especially the reasoning variety like O3 and Gemini 2.5 Pro.

Tina He wrote a lovely, insightful piece on her Substack, and she nails this personal pressure.

When your productivity increases, several mechanisms kick in simultaneously: Leisure’s opportunity cost skyrockets. When an hour of work generates what once took days, rest becomes luxury taxed by your own conscience. Every pause carries an invisible price tag that flickers in your peripheral vision… The game theory is unforgiving: when everyone can produce 10x more, the baseline resets, leaving us all running faster just to stay in place.

It’s this feeling that if AI can do so much for you, every moment you’re not directing it, you’re falling behind. The output potential is so high that the cost of leisure feels enormous.

Somewhat ironically, the frontier model labs themselves, perhaps unwittingly, are encouraging this. Take a look at Anthropic’s “Claude Code: Best practices for agentic coding”. They suggest a workflow:

Create 3-4 git checkouts in separate folders. Open each folder in separate terminal tabs. Start Claude in each folder with different tasks. Cycle through to check progress and approve/deny permission requests.

This is literally parallelizing your work with AI agents, requiring you to manage multiple streams. It reminded me of a short-order cook frantically running between pans on the fire.

I asked on X:

Question is — why do you need human SWEs to carry out these steps and babysit the AI? Or until when?

The trend towards AI working even when we’re not actively directing it is growing. OpenAI introduced scheduled tasks in ChatGPT a few months ago. The AI can be working tirelessly, even without our direct, real-time input.

The folks from Letta AI (aka MemGPT) took this even further with their paper on “Sleep-time Compute: Beyond Inference Scaling at Test-time:

…allows models to “think” offline about contexts before queries are presented; by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time.

So, the AI is pre-processing, anticipating, constantly churning in the background. If you start doing this, the appetite for hardware, for GPUs, is basically going to be insatiable because there’s no real upper bound on how much you could anticipate and pre-compute.

This all reminds me of a recent podcast with Tyler Cowen where he was asked about the bottlenecks to AI progress. His answer was blunt. The interviewer, Dwarkesh Patel, asked, “What are the specific bottlenecks?”

Humans. Here they are. Bottleneck, bottleneck. Hi. Good to see you. And some of you are terrified. You’re going to be even bigger bottlenecks.

So, where does that leave us? In this world where Jevons Paradox at the personal level just seems to squeeze our time more and more, what’s the play? I love how Tina He framed it in the conclusion of her essay:

This may be the good news for those that didn’t dare to fully lean into what they love and want to do. What if the most game-optimal play in the new system is actually to become relentlessly, unapologetically you?

For a quarter-century at MIT, Abelson, Sussman, and colleagues used Scheme to teach freshmen how to manufacture abstractions almost artisanally from scratch. Rooted in the Lisp ethos, the premise was that by understanding a handful of primitives and mastering a small set of combinators, students could build a platonic ladder of abstractions—a veritable cathedral of computation—and still reason about every brick. This pedagogical approach involved first creating abstractions to model the problem domain, effectively designing a custom language in which the final solution could then be expressed cleanly, almost self-evidently.

At that point I’d been programming in BASIC (including its more modern variants like Visual Basic) and Pascal for a few years through middle and high school. The concept of functional programming was alien and jarring to me. But soon it clicked for me, and this way of approaching programming in an almost algebraic way grew on me.

When MIT moved from Scheme to Python for teaching its own introductory CS course (6.001) in 2007 it was news in academic CS circles. It was a signal of the way programming was changing, or already had changed, by the early 2000s.

This wasn’t a sudden coup or a rejection of SICP’s profound ideas. To their credit, Abelson and Sussman themselves had advocated for a change, recognizing that the landscape of programming, and therefore the essential skills required of engineers, had fundamentally shifted.

This transition, like the earlier move from assembly to high-level languages, offers another valuable lens through which to understand the evolution of our craft – another one of which is happening before our eyes today with the rise of AI-assisted coding.

The motivation

As Gerald Sussman himself explained in a 2009 talk:

In 1980, good programmers spent a lot of time thinking, and then produced spare code that they thought should work. Code ran close to the metal, even Scheme – it was understandable all the way down. Like a resistor, where you could read the bands and know the power rating and the tolerance and the resistance and $V=IR$ and that’s all there was to know. 6.001 had been conceived to teach engineers how to take small parts that they understood entirely and use simple techniques to compose them into larger things that do what you want.

But programming now isn’t so much like that… Nowadays you muck around with incomprehensible or nonexistent man pages for software you don’t know who wrote. You have to do basic science on your libraries to see how they work, trying out different inputs and seeing how the code reacts. This is a fundamentally different job, and it needed a different course.

The original SICP approach emphasized deep understanding, achieved by carefully composing systems layer by layer from fundamental building blocks like recursion and higher-order procedures, allowing theoretical mastery of the entire stack.

However, the reality of professional software development had shifted dramatically. It was becoming less about constructing every component yourself and more about integrating large, complex libraries—often opaque black boxes with imperfect documentation. Consequently, the crucial skill evolved from pure logical construction towards empirical investigation—needing to perform “basic science” just to discover how these external systems behaved. To continue teaching only purity and first principles started to feel disconnected; the core task was becoming one of interrogating and orchestrating complex, pre-existing tools, not just building everything from scratch.

Hal Abelson agreed. Both authors had been lobbying to replace 6.001 for years before the final curtain in Fall 2007.

The choice of Python

The replacement course, 6.01, was deliberately centered on the challenges of programming robots to navigate the physical world. This focus was chosen because robots serve as the antithesis of the idealized, predictable components studied in SICP; their wheels slip, sensors yield noisy or inaccurate data, environments change unexpectedly, and all predictive models are mere approximations. Tackling robotics therefore forced students to grapple with a different kind of engineering complexity, emphasizing the need to build systems incorporating robustness, feedback mechanisms, and probabilistic reasoning to handle inherent real-world uncertainty and failure, rather than focusing solely on Platonic elegance.

And why Python? Sussman’s explanation was pragmatic: it had readily available libraries for interfacing with the robotics hardware they were using. This highlights Python’s “batteries included” philosophy – its strength lies in its vast ecosystem of libraries for tackling real-world tasks, even if one doesn’t understand the internals of every library used. The focus shifted towards using available tools to solve practical problems, mirroring the professional landscape.

Lessons for today

The earlier transition from hand-written assembly to high-level languages and compilers is another example of a major change in the way programs were created. It automated low-level machine details via compilers. It raised the level of abstraction, allowing programmers to focus more on logic and less on managing registers and memory addresses. Back then, programmers initially resisted compilers fiercely, fearing deskilling, loss of efficiency and control. But compilers opened programming to a broader audience, reshaping the role of programmers and greatly expanding software’s scope and impact.

Similarly, the transition from Scheme to Python (even in an academic context) marked another profound phase shift, driven by practical necessity as programming evolved from theoretical elegance toward empirical complexity. Today, we’re in the middle of yet another such shift—AI-assisted programming.

Just as Python freed programmers from reinventing wheels, AI-powered coding tools such as GitHub Copilot, Cursor, and Aider liberate us further from mechanical coding details. This arguably pushes the programmer’s role further up the abstraction stack. The “basic science” Sussman described might now involve prompting for, verifying, debugging, and integrating AI-generated code, and focusing even more on high-level design, architecture, and requirements. Tool literacy is beginning to beat syntax mastery.

Yet, just as with previous shifts, programmers today voice similar concerns: loss of control, deskilling, uncertainty around code quality, and fear of diminished roles.

But history offers reassurance: each phase shift in programming hasn’t diminished the profession, but elevated it. Assembly programmers feared compilers but ended up focusing on high-level problem-solving instead of machine instructions. Scheme purists might have lamented Python’s practicality-over-purity approach, yet today’s programmers grapple with much richer complexities, enabled by the vast ecosystems of Python and other modern languages.

Academia is also beginning to grapple with this, with every major CS department scrambling to understand how to teach “Intro to CS/Programming” (and even intermediate courses) when the latest LLM one-shots into the top quartile of the class. Pedagogy must follow practice. If freshmen arrive with ChatGPT open in another tab, pretending it doesn’t exist is as futile as ignoring Python and its “batteries-included” packages in 2007.

Ultimately, each technological leap—from compilers automating machine code to Python’s libraries simplifying integration, and now to AI generating code—has freed human programmers to operate at higher levels of abstraction and tackle more complex problems. The future is programmers amplified.

In one of his missives he makes the case against using natural language for programming:

Formal language/notation is precise! And it has precise rules for composition which enable stating complex ideas clearly.

The virtue of formal texts is that their manipulations, in order to be legitimate, need to satisfy only a few simple rules; they are, when you come to think of it, an amazingly effective tool for ruling out all sorts of nonsense that, when we use our native tongues, are almost impossible to avoid.

Math took off only after it shed the ambiguity of natural language

Greek mathematics got stuck because it remained a verbal, pictorial activity… and the modern civilized world could only emerge … when Western Europe could free itself from the fetters of medieval scholasticism … thanks to the carefully, or at least consciously designed formal symbolisms that we owe to people like Vieta, Descartes, Leibniz, and (later) Boole.

The trap of unknown imprecision. Natural language lets us make statements that sound reasonable, but don’t hold up to rigorous scrutiny.

When all is said and told, the “naturalness” with which we use our native tongues boils down to the ease with which we can use them for making statements the nonsense of which is not obvious.

In other words, a rigorous formal notation forces clarity and precision, which is hard work, but the return is a reliable, dependable and correct system.

Will natural language actually be easier?

a change of interface can easily increase at both sides of the fence the amount of work to be done (even drastically so). Hence the increased preference for what are now called “narrow interfaces”. Therefore, although changing to communication between machine and man conducted in the latter’s native tongue would greatly increase the machine’s burden, we have to challenge the assumption that this would simplify man’s life.

This is a subtle but crucial point. When using any interface between human and computer, there is some work that both parties have to do. For example, with a structured programming language, the human must do the work of converting their thoughts into a syntactically correct program, and the computer (compiler) does the work of translating it for and executing it on the machine.

But the nature of the interface itself also creates an overhead for both parties to manage and interpret the communication happening over that interface. This overhead includes:

• Specification Cost: The effort needed to clearly articulate intent across the interface.
• Interpretation/Translation Cost: The computational or cognitive effort to parse, understand, and translate instructions received via the interface.
• Verification Cost: The effort to ensure the communication was successful and the intended semantics were preserved.
• Ambiguity Resolution Cost: The effort spent dealing with underspecified or potentially misinterpreted communications.

At first glance, the move to using natural language seems like it would drastically reduce the human’s effort (they wouldn’t have to write code anymore!) at the expense of drastically increasing the computation required.

Dijkstra’s argument is that the reduction in precision, the increase in ambiguity, and the resultant increase in the human’s burden to manage that ambiguity and verify correctness mean that the overall human effort could easily increase, potentially outweighing any perceived benefit from not having to learn a formal syntax. (This argument is similar to the one from the “Ironies of Automation”: increasing the automation in a machine can increase the human operator’s cognitive load.)

That’s a pretty damning critique if applied to the vibe-coding style of programming today. He would have absolutely hated vibe-coding.

I agree completely with Dijkstra’s argument when it comes to the advantages of formal notation, but I still think vibe-coding (or AI-assisted coding, which is largely natural-language driven) is a net benefit for building applications today.

Dijkstra is talking about correctness, which rightly was the focus of early computer science. When writing code for algorithms (sorting, graphs) or basic systems (OS kernels, compilers) one needed to be precise and rigorous to get confidence in the correctness of the code implementing the algorithm.

The basic high-stakes layers of computing (CPU instruction sets, operating systems, compilers, interpreters) are fairly settled by now, and were built with a lot of rigor. They are as “correct” and reliable as is possible to make them with human engineering.

(Aside: note that the modern computing stack got quite reliable, but largely without formal proofs. C.A.R. Hoare writes about it here (video), and attributes it to treating programming as an engineering discipline, with thorough reviews and lots of testing.)

But the apps we’re building today do not have a clear notion of correctness. Often “correctness” is discovered during building, testing and releasing the app. Think of social media apps, or CRUD enterprise apps full of tangled biz logic. How would you define “correctness” for such applications? There is a notion of “meeting the specification”, but:

The business specification for modern software is in natural language, not a formal one. Product managers and business owners write their “specs”, such as they are, in natural language. There is no formalism in which to ground the process. That imprecise natural language is taken by system architects and software engineers and turned into software, which does have a formal, executable language.

The specifiers (i.e. product managers or business owners) and the builders (designers, programmers) perform an iterative dance amongst themselves and the application to arrive at the behavior they collectively deem acceptable in their context.

Consider the absurdity of the counter-factual: if modern applications did have a clear notion of correctness, product managers would throw a PRD over the wall to engineering, and would never need to be spoken with again.

The modern activity of software engineering is more about discovery and evolution than building to a static specification. It also largely consists of snapping components together (the long-cherished dream of software engineering). Databases and web frameworks exist–you need to smush your business logic in the middle and build a differentiated user experience.

For such a scenario, AI-assisted coding that lets you quickly build an MVP, quickly change it, and do all that with a natural-language driven interface, is a boon. As folks like Claire Vo and Sahil Lavingia are pointing out, the well-known roles of PM, eng, UX are collapsing into one. An end-to-end “product engineer” can spec, design and build an application, using a fraction of the headcount (usually just themselves!) and time previously required.

The argument for the necessity of formal notation and the benefits of AI-assisted vibe-coding apply to very different layers in the computing stack. Use appropriately.

But if I think about it objectively there are several reasons local LLMs going mainstream might not happen.

Widely used utilities want to be centralized

In the early days of electricity, factories generated their own electricity on-site. This phase was relatively short-lived, giving way to the centralized power grid we have now. The same thing happened with water, trash disposal, manufactured goods and pretty much every staple of modern life.

The Web and its underlying protocols were also motivated by the idea of enabling decentralized control and ownership. But the Web of today is centralized to a few large providers, with billions of users.

There will always be geeks who will host and serve their own email and website, and it is a beautiful thing that that is possible. The protocols are open for anyone to participate in, and web and email servers are open-source and freely available. But the vast majority of people will want shrink-wrapped solutions.

So just looking at the historical trend of how similar technologies have tended to end up with centralized structures (even if they could be decentralized), it’s not looking good for the local-LLMs argument.

Which brings me to my second point…

Centralized management is about labor, not control

The desire to decentralize is often motivated by having control– control over that particular component (electricity, a server, a model etc). But the more important factor is labor.

Owning and controlling a thing takes constant labor. Sure, you could run and maintain your own email and web server. Are you ready for the care and feeding of the surrounding infrastructure? The electricity, the network, the storage? What about configuration, keeping up with updates and security patches? There’s a reason operating the physical and software aspects of running a datacenter is a full-time job.

The same argument applies to local LLMs. It’s rarely about just the LLM. You also need all the surrounding infrastructure, both physical (hardware, network etc) and software (what is the app layer around the LLM?).

The question for you is whether you want to spend your labor on all that, or something else. What is the opportunity cost?

About efficiency and cost

I haven’t seen any rigorous studies making the comparison, but I’d be very surprised if from an efficiency and cost point of view local LLMs beat out a large datacenter that aggregates demand and has economies of scale.

An anecdotal data point: the M3 Ultra Mac Studio with 512 GB of RAM, the current cream of the crop for running large-ish LLMs locally, costs $8549, or $712 per month for 12 months (before taxes). I bet if you sent the collected works of Shakespeare back and forth to the latest and greatest LLM API all day every day for a year, your API bill would be a fraction of that.

Value moves up the stack, away from utilities

If you’re building a mobile or web app, the valuable and differentiated part is the user experience, domain knowledge and the problem being solved. The database or web framework you used is a non-differentiated commodity. It can be swapped out relatively easily.

The same goes for apps built around LLMs. They’re like the jet engines in an airliner: important and necessary, but only a small part of the entire experience and value.

Hotrodders build beautiful cars in their garage, and have a great time doing it. I do want a world where local LLMs are available to the geeks and builders and experimenters to play and build with. I just don’t think local will be the mainstream way they are consumed.

“… but compilers are deterministic, LLMs just generate probabilistic output!”

I think that retort is not even wrong, and I’ll explain why.

The analogy’s central idea was not that LLMs replicate every characteristic of compilers, but that using natural-language-driven LLMs is another large step up in abstraction. Traditional compilers enabled programmers to express themselves in high-level languages instead of assembly. Similarly, LLMs empower programmers to express their intent directly in natural language, translating it into code.

Let’s examine the “but they’re not deterministic!” argument closely:

The entire field of engineering revolves around combining unreliable, unpredictable components into reliable, predictable systems. This principle is foundational in computer systems as well, best described in the classic paper “End-to-end Arguments in System Design”. All modern computers are built on this philosophy, achieving reliability through layered abstractions and error-handling.

Building robust systems with LLMs follows the same approach. Developers can introduce reliability and predictability through techniques such as:

• Writing clear, detailed specifications
• Utilizing tests and functional checks to validate outputs
• Employing feedback loops (including human-in-the-loop validation) to enforce correctness

In fact, many components of contemporary computer systems already exhibit fundamental non-determinism at both hardware and software levels. Some examples:

• Library dependencies: External libraries your code relies upon may update unexpectedly, introducing changes outside your control.
• Garbage collection: Runtime memory management is inherently unpredictable—triggering at different times depending on complex internal states.
• Multi-threading and concurrency: Thread execution order, timing, and context-switching are unpredictable, yet managed through synchronization mechanisms to create deterministic outcomes.
• JIT compilation: Modern runtimes dynamically decide whether and how aggressively to compile bytecode based on real-time data and usage patterns.
• Caching: Cache behavior (hits, misses, eviction policies) is non-deterministic due to dependence on system state and usage patterns.

Modern computing is built around turning non-deterministic components into deterministic experiences for users.

We can—and should—apply precisely the same strategy to LLMs. This is especially straightforward in programming, where we have the luxury of a clear and quick reward function: run the generated code and validate it against intended outcomes.

Rather than viewing non-determinism as a disqualifier for LLMs, we should embrace it as simply another characteristic to manage—just as we’ve always done in computing.

They were showing example after example of end-users – almost everyone at AT&T, even the non-technical staff – using the shell and writing shell scripts to compose complex functionality out of simple programs. The Unix philosophy in action!

This was more than the still-elusive goal of “end-user programming” – it was end-user abstraction! They were building up abstractions, and it was making them more productive!

It’s sad to see the loss of that notion of composability when it comes to GUIs. Modern GUIs just don’t have that compositional layer. You can’t compose GUIs. You can’t script GUIs. You’re just stuck operating them. And it’s not because something prevents that in principle. It was just never a priority for the builders of these giant programs.

It’s funny that after all these decades of GUIs being the dominant computing interaction model, we’re finally “scripting” them with AI computer use models.

The overall philosophy was still one of hand-writing code, but with lots of help from the model. I call this the tactical era of AI-driven coding, based on roughly GPT-3.5/4-class models.

We’re now firmly in the strategic era of AI-driven coding, with agent-modes widely available in Cursor and VS Code, and even on the command-line with Aider, Claude Code etc.

This enables vibe coding, and moving the level of programming even higher in abstraction, encouraging starting with specs. Another mode is test-driven development: “Claude is really good at test driven development, so we often ask Claude to write tests first and then ask Claude to iterate against the tests​”.

The philosophy is to treat hand-editing code (“artisanal hand-crafted commits”!) as a last resort. Context is king! The more context, the better. It could be specs, background info, existing code, library documentation, or even multi-model input like images, video or sound.

The role of the programmer is shifting towards guiding strategic aka agentic coding with the right context.

Of course, evals and benchmarks are an important signal (perhaps the only signal) of raw capability. They show us improvement over time. The fact that many of the original benchmarks from a year or so ago are at or near saturation also tells us a lot.

But if LLMs are the new CPUs, there is an important difference between the way old-school CPU benchmarks (SPECInt, GeekBench etc) were interpreted, and how we should interpret LLM benchmarks.

The translation of CPU benchmarks to performance on real-world apps was objective. A higher SPECInt score, for example, translated to an objectively measurable and valuable metric like transaction throughput or request latency.

Unlike CPU benchmarks, where better performance directly translates to measurable improvements in real-world applications, LLM benchmarks don’t always correlate neatly with real-world utility. Why? Because the success of LLM outputs is often inherently subjective.

Take, for example, a common LLM use-case: summarization. However, in your particular application, summarization might mean something very specific. Maybe you want summaries that emphasize only some aspects of the input, or are always within a certain length range. The metric for success itself is fuzzy. This subjectivity isn’t just a quirk of summarization—it’s a pattern across many LLM applications. That’s why evaluation often relies on human judgment.

There are two common ways of measuring adherence to such criteria: either use LLM-as-a-judge, or human evals. When the developer of the application in question runs a few inputs and subjectively evaluates the output, it is called a “vibe check”. Human evals are vibe-checks outsourced to multiple humans, at scale.

Note that this is not true of all ML models. A fraud-detection or spam classification model might be probabilistic (just like LLMs) but it still has an objectively measurable success rate in terms of false and true positives. It’s just that the kinds of applications LLMs are currently used for anchor heavily on natural language understanding and interpretation, which is naturally resistant to clear-cut objective measures.

If your company or organization is fortunate enough to afford regular human evals on a subset of your traffic, I recommend the following: pick a small subset of your human evals, roughly split between positive and negative judgments. Get your leads together in a room and go through each human judgment. Watch as they all disagree with each other and the human evaluator on the judgment, and as your naive notion that human evals were some sort of golden signal goes down in flames like the Hindenburg.

So what can we do? How can we judge fit-for-purpose and do internal hill climbing for LLM-driven applications?

Embrace the vibe-checks, but be structured about it. Develop a small, representative set of inputs for the particular problem your application is solving. The number doesn’t have to be large– even tens of samples get you pretty far. Then, rather than have golden answers (which are brittle and don’t accommodate change very well), document the properties you’d like to see in the outputs. This should give you a good feel for whether your prompt engineering or model changes or agentic loops are giving you better answers.

Since traditional benchmarking doesn’t work well for LLM-driven applications, we have to lean into subjective evaluation. But instead of relying on gut feelings, we can make subjective evaluation a bit more systematic.

It is one of those pithy quotes that whacks you on the head and makes a bunch of random pieces fall into place and make sense.

Writing is building but software is now content… one of the really cool things is when we we just launched this app called Cora and that has 10,000 people on the wait list. Those are all now Every subscribers.

You can’t write an article that’s going to get you 10,000 new subscribers but you can write software that does it, and that’s incredible.

Let me try to break that down so I can understand it.

“writing is building”

This is speaking to the primacy of the motivation, idea, definition and spec for a product. Perhaps in the case of Every, this also includes the main writing on the site. Most importantly, writing is not content any more.

What is content then? Content used to be a derogatory term. People would churn out lots of it. It would be low-effort. It was used at the top of the funnel. This was the world of content marketing and SEO writ large.

But at the end of the day, this content alone was not executable or actionable. You could not get the content to actually do something for you.

This is where software comes in. Software is utilitarian. You can run it to do things for you, unlike writing. It used to be expensive and hence, scarce. But that is becoming less true by the day in the age of coding LLMs.

In the old world, there would be lots of content at the top of the funnel. This funnel would get fed by social media and SEO. At the bottom of the funnel you’d have some kind of software or service being sold.

In this old world, writing was cheap and software was expensive.

“…but software is now content”

But now that AI makes software cheap, one can quickly build useful software and use software at the top of the funnel. This could not have been done before AI. I think this is what Shipper is talking about when he says that putting out a software product got him way more paying subscribers than churning out articles.

In the new world, the setup looks something like this: you do a limited amount of high-value writing. It may not even be published. But it serves as fodder for what’s next. And what’s next is building lots of exploratory prototypes in software. This software can tackle small niches and long-tail use-cases.

You can do this now because creating software is getting cheaper. When software was expensive you could only build software for the 20% fat head, leaving the 80% long tail to fend for themselves. This software now becomes the top of the funnel. At the bottom, you sell subscriptions or services. This is the strategy Shipper and co are following with Every Studio.

[BTW, AI&I is a shockingly under-rated podcast. If Dwarkesh et al are the “architecture astronauts” of AI, Dan Shipper et al are the miners with dusty faces.]

The early history of high-level languages and compilers is fascinating. It is a tale of the pioneers of computer science charting new territory, but also of deep resistance to new ways of doing old things. And I think it has a lot to teach us about how our tribe of programmers is wrestling right now with the idea of AI and LLMs, writing code.

Early Compilers

In the early decades of computing, programming was done at the most granular level, hand coding in machine code or assembly language. This was painstaking work. Programmers had to manage operation codes and memory addresses directly, which one early pioneer likened to hand-to-hand combat with the machine.

By the mid-1950s, however, a revolutionary shift was underway. High-level languages and their compilers began to automate low-level details, allowing programmers to write instructions in more human-readable forms. Developed at IBM by John Backus’ team, Fortran was the first widely used high-level language. It enabled scientists and engineers to write formulas and algorithms in a more natural mathematical notation, which the compiler would then convert to machine instructions. Fortran dramatically reduced the number of statements required. A program that might have needed 1,000 assembly instructions could now be done in about 50 Fortran statements.

The Fortran compiler quickly proved that compiled code could approach the efficiency of hand-coded assembly. Early skeptics who insisted that high-level code could never be as efficient as hand-crafted machine code were confounded when Backus’ team delivered an optimizing compiler that made Fortran programs run nearly as fast as assembly.

Adoption was rapid. A 1958 survey found that over half of all code being run on IBM computers was generated by the Fortran compiler. This was a stunning validation of this new approach.

Resistance

Even then, many working programmers were skeptical or resistant. They had deep expertise in assembly language and were proud of the arcane skills needed to wring efficiency out of early machines. John Backus later described this culture as a priesthood of programming. In the 1950s, programmers often regarded themselves as guardians of arcane knowledge, possessing skills and knowledge of mysteries far too complex for ordinary mortals.

One of the most frequent arguments against compilers was that compiled code could not be as efficient or compact as handwritten assembly. People would say they could be more efficient in assembly, giving a whole litany of reasons to avoid high-level languages. And this was not entirely untrue. The earliest compilers sometimes did produce verbose or suboptimal machine language. A 1965 Honeywell management briefing noted candidly that a highly skilled assembly programmer could still beat COBOL’s output in efficiency. But it also questioned how many highly skilled programmers are available or even needed at the average installation.

In practice, compiler optimizations improved quickly. IBM’s Fortran team introduced the first optimizing compiler and proved that compiled code could approach or match the performance of hand-coded assembly in many cases. As these successes became evident, the purely performance-based objections began to wane.

The next argument was control. Many assembly programmers were accustomed to having intimate control over memory and CPU instructions. Surrendering this control to a compiler felt risky. There was a sentiment of, if I don’t code it down to the metal, how can I trust what’s happening? In some cases, this was about efficiency. In other cases, it was about debuggability and understanding programming behavior. However, as compilers matured, they began providing diagnostic output and listings that actually improved understanding.

More programmers

There was an implicit fear that making programming easier might reduce the prestige or necessity of the seasoned programmer. High-level languages opened the door for newcomers to write code without years of experience in binary or assembly. The priesthood culture, Bacchus described, suggests that some experts guarded this domain closely.

Grace Hopper encountered this attitude when promoting compilers. Management and colleagues initially thought the idea of automatic programming was crazy, fearing it would make programmers obsolete. Hopper and others had to repeatedly demonstrate that these tools would augment programmers’ productivity, not replace the need for skilled people.

In fact, far from eliminating jobs, high-level languages led to an explosion in demand for programmers. High-level languages opened the door for many more people to become programmers. In the 1940s and early 50s, programming was the domain of a small elite with specialized training, often in mathematics or electrical engineering. By the 1960s, that pool had widened significantly. Fortran had allowed scientists and engineers to directly write programs for their problems, and COBOL did something similar for business. It enabled individuals with domain expertise, such as in accounting or inventory, to learn programming without needing a deep computer science background. As one retrospective put it, COBOL democratized coding. Companies could take everyday people and train them to be useful COBOL programmers in a few months, allowing a new influx of talent into the field.

The result was a dramatic expansion of the programmer workforce. The role was now seen as more akin to logical clerical work rather than electrical engineering. This set the stage for programming to become a mainstream profession rather than a niche craft.

Going higher

As compilers automated the gritty aspects of coding, the skill set for programmers evolved. There was less need to memorize machine instruction sets or calculate memory addresses by hand– tasks the compiler would handle flawlessly.

Instead, the emphasis shifted to problem analysis and program design. This is reflected in the terminology. The rise of the term analyst in job titles during the 1960s. A high-level language freed programmers to think more about what the program should accomplish rather than how at the bit level to implement it. One manager in 1963 noted that with the advent of COBOL, he valued employees who understood the business logic and could structure a program clearly. The compiler would take care of the low-level details if the high-level logic was sound.

In short, the introduction of high-level languages elevated the abstraction level at which programmers operated. By making programming more efficient and accessible, high-level languages enabled many more applications to be developed in a given time. Businesses could automate more of their operations, which in turn drove demand for more computers.

This expanded the scope of automation in society. Entire new domains for computing emerged because software could be written fast enough to respond to business needs. High-level languages were the catalyst that helped move computing from a limited experimental phase into a major industry of its own.

AI

The shift from assembly language to high-level languages with compilers in the 1950s and 1960s has striking parallels to today’s transition from manual coding to AI-assisted programming with tools like GitHub, Copilot, Cursor, Aider, and others.

In both cases, a fundamental change in how code is written is occurring, leading to initial skepticism, gradual adoption, and ultimately, a redefinition of what it means to be a programmer.

The debate playing out today about what it means to be a programmer when LLMs can churn out large amounts of working code is of exactly the same shape. Let’s learn from it and not make the same mistakes.

Sources

1, 2, 3, 4, 5, 6, 7.

There’s a new kind of coding I call “vibe coding”, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It’s possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like “decrease the padding on the sidebar by half” because I’m too lazy to find it. I “Accept All” always, I don’t read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I’d have to really read through it for a while. Sometimes the LLMs can’t fix a bug so I just work around it or ask for random changes until it goes away. It’s not too bad for throwaway weekend projects, but still quite amusing. I’m building a project or webapp, but it’s not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.

“I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works” – this “veni,vidi, vici” for the modern AI age.

This is how a lot of us have been programming for a while already. It will become more entrenched as the models get better. Having to constantly hit “Accept All” will become the number one user complaint.

I spent the weekend vibe-programming a side project I’ve wanted to implement for a while: a time-tracking app that simply regularly looks at my computer screen and infers what activities I’m doing, and whether that’s deep work, light work, or just entertainment. The project itself is not important for this post. I want to focus on the methodology for building it.

I built it using a twist on vibe-coding that I call spec-driven vibe-coding. The overall flow is this:

First, write a spec. Actually, two specs.

First prompt the LLM to take a basic outline of the idea and propose a full product spec. Iterate in Artifacts as necessary, though for me the first cut was pretty much there.

Then give the product spec to o1-pro to produce an engineering design document.

Armed with these two documents, you can zero-shot vibe-code the product. At this point I moved to VS Code with agents. I put the above two docs into a “docs” folder in my repo, attached it as context in Copilot Chat, and simply asked it to implement the spec.

In this case, I also prompted it again to write tests to cover all the code it had generated. It took a few rounds of back and forth to get all the tests to pass. Then a few more rounds to resolve runtime errors to get it to run and actually work. All in all, this was about a Saturday afternoon’s worth of combined time. From 3-4 bullet points of the basic idea to product spec to software design to running code.

And then I started adding features. I kept the same basic principle in mind: first write a spec, then update the software design doc, then ask the LLM to write code that meets the spec and the design.

Throughout the entire process I did not write any code by hand. I found bugs and issues, yes, but I gave the stack traces or test failures back to the model to solve. I was getting tired of hitting “Accept”.

This is the worst this capability will ever be. This was a tiny weekend project. (Actually, if I had to do this by hand before the AI age, I’m sure it wouldn’t have fit in a weekend, and hence, would never get built.) This basic flow will work for larger and larger codebases over time. Raw-dogging code by hand will have the same frequency as writing assembly.

If you are a programmer, you should be embracing AI to increase your impact and agency.

Appendix: Prompts and Results

Prompt to take basic outline of idea and produce a product spec:

Canvas help me write product spec for the following idea. I am also attaching a diagram describing the flow I’m thinking of. This product spec will then be given to a software engineer to design and implement, so it should have enough detailed description of the product for an engineer to design and implement it fully.

Idea for Time Tracking

#quantified-self

### Goal Automatically measure which apps and tasks I’m spending my time on, on my computer

### Outline

• Take screenshots of desktop every ~1min (configurable time interval)
• Have a vision LLM describe that image
• Then do some more summarization and analysis on that
  • This should give me a picture of what I’m spending my time on

Resulting product spec.

Prompt to take product spec and produce a detailed engineering design document:

i will give you a product spec below, as a markdown file. thoroughly read it and then produce a software design document to implement that spec. this design doc will be handed off to a programmer for implementation, so it should have enough detail that the programmer can implement it without any ambiguity.

basic tech choices: this will run in mac os x, should be written in python, run as a command line app (no gui necessary).

<paste in product spec here>

Resulting design doc.

When an analyst presents a financial model, does he mention that it was made with the help of Excel?

Whether the tools in your arsenal are word processors, or grammar and spelling checkers, or human editors, or modern LLMs, you, alone, are the author.

Ethan Mollick’s proposed policy is the most sensible one for the AI age: use AI all you like, but you are responsible for the work turned in. Your name is on the paper. You get the grade.

This applies to AI slop too. AI isn’t responsible for slop. There is a human author behind it. They should put their name on the slop. It should have all the same consequences as putting out purely human-written poor work into the world.

In theory, Emacs should be the ideal UI for the age of LLMs, because after all, it’s all just text, and Emacs excels at manipulating text. But!

The first thing that pulls me out of Emacs is programming with Github Copilot, and its deeply integrated experience with VS Code. Some of that interaction is probably in closed protocols between VS Code and Copilot backends, but it is also the bespoke UI for inline chat and multi-file edits.

The next thing that pulls me out of Emacs is prototyping and writing and iterating with Artifacts and Canvas, and the general UI affordances in the bespoke clients for the major LLM providers. This holds for both code and prose.

Note that the above arguments have the same shape as those from a decade or so ago that claimed (rightfully) that crafted IDEs offer a richer, smoother experience for programming (particularly in IDE-dependent languages like Java) than trying to spend hours recreating the same functionality in Emacs only to end up with a clunky experience. That might still be true to an extent but the magnitude of the problem is a lot less severe due to a system-level innovation: language servers that speak a common, broadly supported protocol (LSP), and every language toolset shipping with one that was editor/IDE-agnostic.

AI dunkers are doing newbie programmers a huge disservice. Those who listen to the AI dunkers and internalize their backwards-looking, risk-averse, techno-primitivism will be hurting their learning, growth, and ultimately, their careers. While they’re also doing a disservice to programmers well into their careers, I want to focus on the learners and early-career programmers in this post.

The AI dunkers are only the latest in a continuous line of techno-primitivists.

When compilers abstracted away assembly language and allowed programming in high-level languages, techno-primitivists argued about the loss of control, the loss of performance, and the loss of understanding of how the machine works.

When IDEs and auto-complete made it easier to write lots of code quickly, techno-primitivists argued about the loss of “true” understanding of the language and libraries programmers were using, about writing expedient code, and extolled the virtues of the spartan steel-minded programmer writing code “raw.”

And now, at the dawn of the next step-function in how we write programs, the techno-primitives stand athwart the gates, warning us of the perils ahead and the virtues of past ways.

What should those who are learning programming do instead? They should embrace this new way of programming but do so while prioritizing learning, understanding, and growth through challenge. You have agency– use it! You are not a slave to the tools before you.

First of all, don’t use AI to get answers and move on. Use it to learn and understand. Imagine having a pair programmer mentor with infinite patience and deep knowledge. Ask it to explain existing code. Ask it to review the code you wrote. Ask it for idiomatic ways to rewrite your newbie code. Ask it for five alternative ways to write the same thing and what the tradeoffs are among them. Ask it which standard libraries you could’ve used instead.

The message should be one of encouragement and exploration of these new tools, fully embracing them to become more effective programmers, with a minor cautionary note about their failure modes and pitfalls—not one of dismissive skepticism.

I’m currently an engineering manager at Emergence AI. Before that I was at Google in various roles (SWE, TL, TLM, EM). Most of my professional work is organizational and technical architecture/design, with a smattering of coding. I do code regularly for personal tools and explorations.

Programming

I use LLMs for programming a lot. It is now a basic tool, on par with having an IDE. I sometimes see techno-primitivist takes on how using AI/LLMs for coding is “bad”. I can only hope that folks getting into programming now aren’t swayed by that.

My two main tools are: Copilot in VS Code, and aider on the command line.

I’m a happy GitHub Copilot subscriber. Pretty much all my programming these days is in Python, and it works great for that. Just like the writing use-case, having Copilot means never having to start from a blank slate. I’ve been writing Python for more than two decades, so I have an eye for when it produces code I don’t like (it almost never produces flat-out wrong code). Depending on how lazy I’m feeling, I either edit it myself, or simply re-prompt it. I exercise the full range of Copilot features – the core feature of tab-completion (usually based on a comment or a prompt), as well as its chat features, for asking general programming questions, reviewing code and explaining code. Using “/explain” very often saves me a round-trip to Google and library or package documentation.

But when I need some very heavy lifting done, particularly for project-wide edits across multiple files, I turn to aider. Aider with Claude-3.5-Sonnet is a very powerful combination. Things for which I turn to aider:

• scaffolding out a new project with basic functionality working (fastapi, uvicorn, gunicorn etc)
• generating tests from scratch
• reviewing: take a messy first pass, prompt it to be a thorough reviewer and stickler for idiomatic usage, and rewrite it to be clean and modular
• refactoring across multiple files. Works great if you first generate tests.

Aider reports the cost of a session based on token usage (you have to bring your own API key for the underlying model and pay for it), and it is heart-warming to see the number of times it has generated code for me that would’ve taken me 30-90 minutes and it cost something like 7 cents.

Note that Copilot and Aider represent two very different modalities for programming with LLMs: copilots and agents. Copilot, is, well, a copilot. You, the programmer, ask it to write or re-write code, and it does that in the same place where you do your edit-compile-run cycle, namely, the IDE. Aider, on the other hand, is outside your IDE. It is somewhat more autonomous. It runs in the terminal. It performs agentic workflows and often performs multiple steps (each using LLMs) to carry out the programming task you ask. With tools like Cursor, these two modes are starting to unify.

Media consumption - reading and watching

I read stuff on the web and watch videos on YouTube. I’ve written a couple of utilities for myself to save time on both.

• HN summarizer: Given an HN story ID, fetch content of the article it is talking about as well as the comment thread, summarize it.
• YouTube video summarizer: this one is more complex. I wanted to do something better than a naive video summarizer, which would transcribe the video, then simply prompt an LLM to summarize it and present a textual summary. I wanted a video summary that was edited from the original video. So I used whisper to transcribe the video with word-level timestamps, then prompt an LLM to construct a summary consisting only of direct quotes from the transcript, and use those two combined to output a chopped up version of the original video. Works great. I use it daily on multiple videos, usually long lectures or podcasts.

Another common workflow is quickly understanding research papers. I’ll throw a PDF of the paper into the ChatGPT desktop app, then interrogate it. It helps me quickly understand the key findings, assumptions and methodology in a paper. This is somewhere between just reading the abstract and doing a quick page-by-page read.

Since the introduction of ChatGPT Search it has also significantly eaten into my Google search volume. I now trust web-grounded RAG enough for many day-to-day queries.

Writing

Two main themes here: going from unstructured to structured, and never starting from a blank page.

I will sometimes do an unstructured stream-of-consciousness braindump (could be either typed-out text, or just spoken voice into ChatGPT’s advanced voice mode) and ask the model to rewrite it cleanly. Helpful for capturing crisp summaries and action items from complex discussions.

When writing (design docs, requirements, blog posts), get over the activation energy of starting from a blank page. Ask the LLM for an outline, or use it as a Socratic partner to flesh out ideas or pressure-test them.

Personal utilities

Miscellaneous use-cases and personal utilities that don’t fit neatly in the above buckets:

• create a math quiz on a given topic for my elementary school kids
• given a transcript of a video I’ve recorded for my YouTube channel, suggest chapter titles and timestamp markers
• trawl over my screenshots directory (where every filename is of the form “Screenshot-<date>.png”), then use a combination of a vision model and language model to give it a short, descriptive filename. From chaos, order!
• create a chatty TODO tracker. Braindump a bunch of things I need to get done, get a clean, itemized TODO list. Then conversationally tell it as I get things done. I can ask things like “what’s left to do?” or give it natural language updates like “move item X to tomorrow”.

Huge shout out to Simon Willison’s excellent llm command-line tool that makes it a breeze to invoke pretty much every LLM from the command line, and do so in a unix-tool way that can be composed with pipes and all. It also has a clean, simple Python API that has become my goto library for abstracting over vendor-specific LLM client libraries.

Barriers and wishlist

As you can see from the above, I’m a heavy AI/LLM user, and I derive a ton of utility from it. I’d like for that to be even more frictionless. Here are my major barriers to even deeper usage of AI (or a wishlist):

• deployment: this is my biggest pain-point. I can quickly whip up a Python script that strings together some LLM calls and does some local file I/O. But that only works on that local machine. What if I want to wrap it up as a web app, living in the cloud, so that I can use it even if I happen to be working on another computer? Then I immediately land in the quagmire of containerization and fighting with AWS/GCP etc to get it running in the cloud. I wish there was a quicker, easier way to wrap up these things for cloud deployment.
• prompting: seems funny to say, but prompting is still a bit of a black art. I have to beg the LLM gods. There are magic incantations like “take a deep breath”. There is a lot of trial and error. Part of this is clarifying my own thinking so that I can instruct the LLM clearly about what exactly I want. But there still is a bag of tricks one must know.
• background running: related to the deployment point above, but there are some use-cases I have in mind where I’d like to trigger LLMs with tasks periodically or based on a condition. This is essentially hooking up scripts to a cron schedule or other trigger, but it comes back to the pain of deployment.

How I feel about it

I’ll confess, for all my techno-optimism, I was taken aback when GPT-4 was released, and they emphasized how good it was at coding. I wrote back then:

The release of GPT-4 yesterday and the subsequent demo has made me feel a mix of excitement, melancholy and disorientation.

A couple of years ago I thought I had a decent grasp of the trajectory of programming as a field, and as a career path. Languages, frameworks and libraries would get better at modeling and abstracting more complex programs, safety vs performance would be less and less of a conflicting tradeoff, compilers would get better, IDEs and tools would get better, and underneath it all, even if Moore’s Law was hitting a wall, we’d just continue throwing more cores at larger problems. But at the end of the day, the immense cognitive leverage that comes from encoding something into executable software would ensure that programmers were always in short supply. But now I feel unsure and disoriented, not certain about where I stand.

I’m beginning to get more and more convinced of Matt Welsh’s take on how LLMs spell the end of programming as we know it (YT video of his talk). The last few decades saw steady linear improvements in programming languages and tooling. The next few years (who knows, maybe months?) are going to be uncharted territory, a step change. Like the saying goes – may you live in interesting times.

More than a year and half later, after having used these tools almost daily, I couldn’t be more optimistic. Don’t believe the doomers and naysayers! As Ethan Mollick likes to say, “Always bring AI to the table”.

TL;DR: I disagree with the article, but I want to make it clear that this isn’t a critique of the author. I’m more interested in diving into the ideas and arguments because I feel it reflects a lot of the anti-LLM (Large Language Model) sentiment we see today, particularly in the programmer community.

So, let’s dive in.

The Common Argument Against LLMs

Right at the top of the article, the author concedes that they’re making a stronger version of the common argument against LLMs. Now, the usual argument against using LLMs for programming is that they’re not as good at generating code as some claim. It’s all hype, they say.

People who claim massive productivity gains often point to how quickly they created an app using an LLM. The critics argue that these folks are just blowing smoke. Sure, there’s room to debate where current LLMs are good at generating correct code and where they fail. That’s a fair discussion. There are certainly areas where an LLM won’t be of much help—like when you’re trying to code an entirely new algorithm.

But, of course, there are other areas where LLMs can be a massive productivity boost.

A Stronger Rejection of LLMs

Here’s where the author steps things up a notch. They’re not just saying LLMs aren’t useful. They’re going further, claiming that even if LLMs were good, they still wouldn’t use them.

This is a much more personal, almost aesthetic, stand against LLMs. That’s what I found interesting.

Programming as Art vs. Commercial Endeavor

Now, we get to the meat of the argument. The author treats programming as an art form, stating that programmers have an infinite canvas to express their creativity, which makes programming a form of self-expression. I think this is the crux of a lot of anti-LLM sentiment in the programming world.

I also think they’re conflating two separate things. One is using programming as a means of self-expression, like the author is saying here. The other is using programming as a means to an end—to accomplish a task, run a business, make a profit, and so on. These are two very different endeavors.

Programming as an artisan or an artist is a personal activity. Programming as an industrial or commercial activity is about efficiency, productivity, and scale.

If we’re discussing how LLMs can or can’t help, I think it’s crucial to separate these two types of programming.

You Are Not an Artisan

This reminded me of an old blog post by Venkatesh Rao from over a decade ago, titled You Are Not an Artisan. It’s a fascinating read, and I recommend it, but let me highlight some relevant parts here.

The piece talks about how modern workers often view themselves as artisans, even when working in industrial or commercial contexts. One key point in the article is about how artisans confuse “economically essential variety” with “economically optional variety.”

To paraphrase: if uniqueness in a product mainly makes the producer feel special but doesn’t lead to profitable differentiation, it’s optional, like latte art. In contrast, economically essential variety must be handled to make a market profitable.

The author of the Hacker News article seems to be approaching programming more like the former—as an artisan craft—while many of us in the LLM space see it as the latter: a commercial activity that’s about solving problems and achieving goals.

The History of Techno-Primitivist Thought

There’s a lot of what I call “techno-primitivist” thinking in these arguments. Throughout the history of programming, there’s always been a nostalgic school of thought that resists new developments and clings to older ways of doing things.

The last big wave of this was in the 90s and early 2000s when IDEs (Integrated Development Environments) were gaining popularity and replacing plain text editors. There’s a famous article from that time called Does Visual Studio Rot the Mind? by Charles Petzold (2005). In it, the author rants against IDEs, arguing that tools like IntelliSense (autocomplete) sap our programming intelligence.

Sound familiar? It’s the same kind of sentiment we hear today in the anti-LLM crowd.

In both cases, there’s a yearning for the past—for a time when there were no APIs, classes, properties, or IDEs—just the programmer and the code.

Programming Tools Throughout History

The author also makes an interesting point comparing using LLMs to asking an artist to paint for you. But if we look at the history of art, this isn’t so clear-cut. Take conceptual art, for example.

One famous case is the work of American artist Sol Lewitt. His wall drawings were often executed by other people based on high-level instructions he provided. So who’s the real artist here—the one giving the instructions, or the people executing the work?

In programming, too, this raises the question: if I give an LLM instructions and it writes code for me, am I still the programmer? I would argue that even if you’re approaching programming from an artist’s point of view, you should use all the tools at your disposal to push your artistic vision further.

The March of Abstraction

The entire history of programming is about automating and abstracting away lower-level tasks. We’ve gone from binary code to assembly, to compilers, to higher-level languages, and beyond. Each step involves creating black boxes to simplify our work.

Using LLMs for programming is just the next step in this evolution. These tools allow us to tackle much larger, more complex tasks than we could have done before.

My Personal Experience with LLMs

Personally, my experience with LLMs has been overwhelmingly positive. I’ve used them both for work and for personal projects, and they’ve saved me tons of time. I’ve been able to complete projects that wouldn’t have fit into a weekend before, but now they do, thanks to LLMs. It’s been awesome.

To borrow a quote from Don Draper in Mad Men: Always look forward.

I hope this gives you something to think about. If you’re pessimistic about LLMs, I’d love to hear why and learn about your experiences. If you’re already using LLMs to speed up your programming or explore new things—welcome to the party. It’s the wild west out here, and we’re all figuring out what’s possible with these tools.

Is Copilot is Making Programmers Worse?

The article is titled “Why Copilot is Making Programmers Worse at Programming,” and right at the top, the author acknowledges that there’s a trade-off here. These tools are really helping programmers increase their productivity, but as he says, there’s a trade-off because he’s concerned that they may have unintended consequences on the quality and skill set of programmers.

Eroding Core Programming Skills?

The first argument he makes against using AI for programming is that it can erode core programming skills. If you have the AI write code for you and you don’t think through the problem yourself, you’re gradually going to lose that skill. You’re going to gradually lose the ability to use critical thinking and deep reasoning.

There’s some truth to this, but if you look at history, there has been a long series of such arguments going all the way back to Socrates. Socrates didn’t like the idea of writing because he thought that if we didn’t hold these ideas in our heads and repeat them by speaking them out loud, we would forget them and not understand them as well. So, he did not like this idea of externalizing thoughts by writing.

Now, I don’t find this particular argument very compelling because it depends on what you mean by core programming skills. Would you have said that programming without an IDE is a core programming skill? That you should not have to rely on all the conveniences of an IDE? You could go lower and lower down the ladder of abstraction. Would you have said that assembly language is a core programming skill? I don’t think anyone misses programming in assembly today.

So, I think this is a matter of abstraction. Even if you have a new tool that is now writing the code for you, all that it means is that there’s a layer of abstraction on top of it—just like compilers compile your high-level language code down to assembly and machine code.

Over-Reliance on Auto-Generated Code

The next argument is really interesting: over-reliance on auto-generated code. I think there is a lot to this point. I like to make the parallel with how airline pilots use autopilot. There has been a ton of literature about how automation dependency, because of airline pilots using autopilot, has caused their flying skills to atrophy over time.

So, there’s a fine line to walk here because you definitely want to get the benefits of these new tools. They allow you to do more, faster. They allow you to build faster. Just like overall, autopilot really had a drastic influence on improving the safety of air travel, but it does come with this problem. It comes with this problem of automation dependency and pilots forgetting the fundamentals of how to fly.

So, this is somewhat related to the previous point, and I’m not sure what a good solution to this is, other than simply, as some have put it, disengaging the autopilot from time to time. So, programming without AI from time to time just to see if you can still do it, just to keep your skills sharp.

Lack of Ownership and Responsibility

The next point is about lack of ownership and responsibility. The author says that if you, as an individual developer, don’t stand behind the code you write, it might lead to sloppy or buggy code. After all, AI-generated code could have bugs or could just be flat-out incorrect.

This is definitely true now, but I think it will be less and less true as time goes on. It also depends a lot on your domain. If your domain is well-trodden, like building a web app backed by a database, my guess is that the error rate of LLM-generated code in that kind of well-understood domain is rapidly going to get to zero. In other domains that are not as well-trodden, the human programmer will still have a big role to play.

The other point I want to make here is that, in at least an industrial setting, if there are bugs, you should look more at the overall process rather than at individual programmers. This goes back to the idea of blameless post-mortems, which started at Google SRE. The main tenet is that when a system breaks, when there’s an outage or a bug, do not blame the individual teams or programmers. Instead, look at the root causes and the process that allowed such a bug to pass through undetected.

The same thing could happen with programmers who aren’t using these tools either. So, at least on this point, I think it goes back more to the process than to the individual programmers.

Reduced Learning Opportunities

Now, this is actually the one point in this piece that I disagree with most strongly. I feel like it’s exactly the opposite: LLMs lead to way more learning opportunities than before. They democratize the learning of programming more than anything else I’ve seen in two decades of programming.

Why is that? Because you can use the LLM to learn programming. You can use the LLM as a tutor to help teach you programming. If you’re a programmer, you can use the LLM to explain code to you, to review code for you, to suggest alternatives to code that you might have written. And if you just use the LLM as a co-pilot during the act of programming, you can learn a lot, especially if you’re new to a language and don’t know all the idiomatic ways of expressing common things in that language.

Narrowed Creative Thinking

Programming is as much about creativity as it is about logic, says the author. For this one, I actually talked about this specific point in my previous rant about creativity and programming with AI, so I’ll just leave a link to that in the description. There are interesting arguments both ways, so go take a look at that video.

Dependency on Proprietary Tools

There’s definitely something to this, right now, but the dependency on proprietary tools is rapidly decreasing because of all the open-source models—things like Llama, Phi, and Gemma, and so on. Right now, they’re not up to the level of the proprietary models like GPT-4, Gemini, or Claude, but the gap between open models and proprietary models is rapidly shrinking. I think very soon the open models will be good enough, if not just as good, so this point is gradually going away.

A False Sense of Expertise

A developer might feel proficient in programming because they quickly generate working code with the help of Copilot, even if they don’t fully understand the code.

I think this is mitigated a lot if the developer has the right attitude. If the developer just uses Copilot-generated code without understanding it at all, yes, they can do that—they won’t learn much. It’s similar to a high school or college student using an LLM to write an entire essay and just turning it in. But if instead, a developer uses the LLM to actually explore the problem, explore the solution, have a dialogue with the LLM about alternatives and about understanding the code, I think that might actually help them. Like I mentioned a couple of points ago, it might actually help them learn more, understand more, and develop more expertise.

Conclusion

To wrap up, I think the author makes some good points. He’s looked at this problem from a lot of different angles, and this debate is still going on. I myself have a bias towards being a heavy adopter of these tools. I think you should embrace them if you’re a programmer. There’s no going back. These tools will only get better over time, and they’ll only be adopted more and more over time.

So, if you, as a programmer, want to stay relevant, you have to realize the risks and shortcomings of these tools but not ignore them. You have to embrace them and learn them to the best of your ability—just like a couple of decades ago when IDEs became mainstream, and you had to learn how to wield your IDE in order to be a productive, modern programmer.

I frame it as follows: in an AI interaction there are three parties–human, AI, artifact; which of them is the subject, which is the object and what is the verb or action relating the two? It is a straightforward application of the same concepts from grammar. A secondary aspect of this framing is the degree to which the verb is constrained.

Let’s look at a few concrete examples:

Generic chat: probably the most dominant modality right now. Human comes to ChatGPT (or Claude, or Gemini), stares at an empty text box. The verb in this case is completely unconstrained, depending entirely on the prompt provided by the human to the AI. What about picking the subject and the object? In a typical scenario, the human will act as subject, initiating the action. The object will be the thing or goal the subject wants out of this interaction. This, by the way, is the reason why the artifacts feature in Claude is groundbreaking. It clearly separates the object of the conversation as a distinguished entity.

But while the human acting as the subject is the most obvious path, there are many valid and useful scenarios where the human is the object.

Consider conversations where the AI is performing the role of a tutor, mentor or coach. Even though the human initiates the conversation, he is the object. The human is being taught, or mentored, or coached. The AI is the subject, performing that action upon the human.

The arc of improving the UX of AI seems to be one of constraining the verb and making the object explicit. Prominent examples are using Gemini in Google Workspace and Apple Intelligence. The way the AI is situated and presented makes the artifact being worked on (the object) readily obvious. It’s the document or email or spreadsheet you have open. The verbs are also constrained to some degree to common actions: summarize, rewrite, adjust tone, suggest formulae. Where there is free-form chat, the UI nudges the user to understand that this chat is within the context of that artifact, nudging the user towards asking questions about that open document or sheet, not generic free-form questions.

AI for developers, with products like GitHub Copilot, falls somewhere in between. The subject is almost always the human programmer. The object is almost always the code being worked upon. The verb is a combination of constrained and free-form. It is constrained when seeing inline completion suggestions: there is implicitly only one verb of “complete what I’m typing”. But the other half of Copilot’s feature set is Chat, which is open-ended. But chat has pre-defined verbs in the form of command like “/explain” or “/refactor”.

When looked at through the lens of constraining the verb, we see a replay of the age old CLI vs GUI evolution. A free-form CLI with little to no constraints on utterances is meant for experts and daunting to most users. For the masses we have a crafted and constrained UI with pre-determined happy paths and constrained verbs (actions) that the user can pick from a given menu of options.

If the user is typing in a prompt, they are at the AI CLI. If not, they’re likely using a product UI.

This parallels the days before compilers– when you had to write bespoke assembly for each platform. Software was not portable.

Later, compilers for high-level languages could target various ISAs without the programmer having to worry about it.

What if– LLMs came with “atomic prompts”? Prompts tuned for that specific model for basic tasks like summarization, simple reasoning, personas etc. Make it part of an (extended) model card. Would certainly take a lot of guesswork out of porting prompts across models.

And while we’re at it can we please also make it so that we never have to say “Take a deep breath” and “I will tip you $200 for getting this right”? That would be great.

The programmers at the dawn of computing were nerds. They had no abstractions. They dealt with raw hardware – switches, circuits, currents. Binary code. Punched cards. Assembly language was a respite.

The next step up were high-level languages like FORTRAN and C. Imagine having mnemonics like FOR and variable declarations, after futzing with assembly.

The arc of high-level languages was a long one. Of course, the languages, theory and tools have taken massive leaps over the last few decades. But they’re still languages, still authored by and debugged by human programmers.

The people who found this activity appealing were nerds. They were attracted to the unforgiving yet clean logic, the relentless detail, of the machine. Even after hitting the depths of despair during debugging, emerging on the other side was cathartic because while the solid ground of reason was obscured, you managed to find it! And most importantly, it existed.

The platonic universe of pure reason was manifested in a bunch of circuits, and your messy utterances to make the current dance through them. It was solid ground, unassailable, its rewards and comforts showered upon those willing to endure the hardship of – first, thinking clearly and logically, and second, expressing it in executable form.

This is what the programmers always had over the mathematicians. Math was pure abstraction and reason, and it was beautiful, sure – but programming was all that, and it was executable. It could run! It could talk back to you, reflecting back to you the rigor or sloppiness of your own cognition.

All this is to say that computers, and “talking” to them via code, was for nerds. Those who felt more comfortable with things rather than people. In fact, they avoided people. People were unreasonable, unpredictable, and inconsistent.

The activity of programming has become easier – dare I say, friendlier – over the years. The languages have soared higher and higher, away from the gritty details of the machine. The tools have become more approachable. We want kids to code, and they seem to be doing that pretty well.

Hardware has become cheap and plentiful. Plentiful enough to support higher and higher level abstractions. The languages aim to model not the machine, but your business, and your problem. The words in your program are from your domain, not from the machine. This is all progress, and the modern economy is built on it.

This has reduced the need for the monastic mental rigor needed when the field was more primitive. The field has been de-nerdifying over time.

Which brings us to large language models and prompt engineering.

Programming languages became higher and higher level… until they become natural language. That seems like a natural step, expected even. This step completes de-nerdification of the field.

The process is in its infancy, and will probably take a while to play out. There are also plenty of people who believe that this is a blip, as any nascent change appears to be. But it is inevitable.

But for the nerds of programming, the problem with building a system out of prompts is… that you have to coax the machine into doing what you want. In informal, natural language. Just like people.

A programmer has to program by talking to the computer as if it were a person. Just like the Star Trek computer. Yay!

But the nerd has a nostalgic feeling of loss at this impressive achievement. Someone called this feeling vesperance: “the solitary emotion of wistful recognition of the present as a fading era, tinged with anticipation for an unrecognizable, transformative future.”

A key point missing in the discourse around “is AI going to make coding redundant?” is that there is a hierarchy to programmers, and there is a hierarchy to the difficulty and ambiguity in programming tasks. It seems obvious to state, but: AI/LLMs will attack the easier side of that spectrum to start with, and in fact, they already are.

I predict that entry-level junior software engineers are going to get eaten by large language models that can generate code.

Now my take is actually a moderate one. If you listen to folks like Matt Welsh of Fixie.ai, he has a much stronger position. He has the position that all software engineering is going to get eaten by LLMs and AI. And maybe that’s true, but I’m making the softer claim that it’s going to happen more immediately and more obviously for entry level positions really soon.

We have to start by first understanding the level progression of software engineers. If you go look at levels.fyi, the levels are similar across most tech companies. The first three levels are L3, L4, L5. L3 is entry-level software engineer. That is usually people new to software engineering or recent graduates. L4 is mid-level. L5 is senior. The general scope and responsibilities for an entry-level L3 software engineer are to design, implement, debug, and deploy software that is fairly well-understood and well-defined in scope. Usually this takes the form of someone a bit more senior like an L4 or an L5 doing the higher level or more complex design for a system and breaking it down into well-defined tasks. Then L3 software engineers do maybe some more detailed design, and the more well-defined and well-scoped implementation tasks to implement that larger system.

Basically, what an entry-level software engineer does is take well-defined, well-scoped tasks and then be proficient enough in coding, debugging skills to quickly, correctly, cleanly implement them and deploy them to production and to be able to debug those things if need be. And then once you get to higher levels like mid-level or senior level, it’s more about being independent. It’s more about larger scope, larger ownership, leadership, all that stuff. But today we’re just talking about entry level software engineers.

So given that description of what an entry level software engineer does, it’s not that hard to see how the code-generating LLMs of today are creeping very close to it. And if you wait another maybe year, which is practically infinity in machine learning time scales, that it would be pretty easy for LLMs to accomplish that. With context windows getting larger and larger, you can fit more and more surrounding code, adding more context so that you can ask the LLM to write code for something that is very specific to your system and to your code base, your module, your implementation and generate code that understands what the surrounding classes and methods and abstractions are doing and actually uses them as opposed to writing generic code that uses publicly available APIs that the LLM has already been trained on.

If you look at recent increases in context window lengths, for example, Gemini 1.5 went all the way up to a million tokens, and even larger sizes in research. And a million tokens is pretty huge. A million tokens is 750,000 words. A lot of even medium-complexity GitHub repos could fit entirely within that window. Now, if you’re working in a closed source enterprise code base, it’s probably a lot larger than that, but a million tokens might still be enough to fit the module or the subsystem that an entry-level software engineer is working on. And you can put all that into the context window plus your instructions for what you want to do.

Today the “hit rate” for accomplishing a task right away is not high. Often a human software engineer has to take that first draft and tweak it a little bit to make it work.

The writing is on the wall. What are the implications of that? Are you just going to have senior software engineers do design, spell out architecture, and drive the LLM, so to speak, give the LLM direction, look at its output, nudge it in the correct direction?

It gets a bit more speculative and vague as you go up the stack from junior to mid-level to senior and beyond. But at least for the kinds of things that entry-level software engineers do, I don’t think it’s a far stretch at all to say that code-writing LLMs can do that kind of work. They can do a lot of that kind of work even today, and very soon they’ll be able to do most, if not all of that entry-level work.

The open question is what that means for the overall hiring pipeline, and career development more generally. If you just skip this entry-level step in career development, how does one even get to mid-level or senior level? How do you get the experience needed to reach those levels? It becomes a bootstrapping problem. And I don’t have any good answers to that. I don’t think anyone does. I think all this is happening so fast that everyone is pretty much making it up as they go along.

That’s an important question for computer science or programming graduates coming into the market, into the tech hiring market in the next few years. They’re going to need a lot more than coding and implementation chops to stand out in the hiring pool. And once they do get hired, again, they’re going to need more than coding and implementation chops to stand out among their colleagues.

At the very least, what’s clear is that if you’re graduating now or soon, you have to be extremely familiar and extremely fluent with AI-assisted programming. You have to understand how to prompt an LLM, how to prompt an LLM to get specific kinds of code, how to prompt an LLM to help you debug, how to prompt an LLM to help you critique your code, review your code, debug your code. You have to know various ways of prompting LLMs. You have to know how to work with LLMs. And I think that skill set will become as basic as knowing how to use an IDE or knowing how to do a edit-compile-debug cycle.

I used a prompt that tells the LLM to generate a relevant follow-on question, then use that question as the next prompt in the chat.

You can repeat this for as many turns as you want– depending on how long you want to listen, and how much you want to rack up your API bill.

Below is an example of GPT-4 talking about the Dylan Thomas poem “Do not go gentle into that good night”. I’m no lit major, but sounds like a decent exposition of the poem.

The prompt to generate follow-on questions was:

Given the above what is one follow-on question that would help me learn about this topic? Please reply with only the question, without any leading or trailing information.

In their most general sense, compilers are translators from one language to another. We usually narrow that meaning to only allow translators that output low-level executable machine code, but a program (like an LLM!) that took natural language and emitted executable code in a high-level language could also very well be classified as a compiler.

In Large Language Models: Compilers for the 4th Generation of Programming Languages?, the authors chase exactly that line of reasoning:

…this is a speculative paper discussing whether large language models could be considered a higher level of programming language in relation to current high-level languages. In short, assembly language (2nd generation) replaced punch-card programming (1st generation) by introducing mnemonics. These allowed larger and more complex programs to be created in less time. High level languages (3rd generation) in turn replaced assembly language by introducing structured English constraints. The hypothesis explored in this paper is that large language models could be a 4th generation language, replacing high-level languages by allowing natural language specifications.

The most obvious objection to this line of reasoning is that compilers are deterministic and (modulo bugs) will emit “correct” code, whereas LLMs are probabilistic and will sometimes emit erroneous or incorrect code. The authors propose that feedback be used as a central mechanism. Syntax errors should be fed back to get correct code. Code that does not adhere to a spec (as expressed in unit tests or other checking mechanisms) should be fed back to improve the spec, i.e. the input prompt.

It can also be argued that the deterministic translator is primarily concerned with syntax (i.e. the structure of sentences based on the Chomsky hierarchy) and the probabilistic translator with semantics (i.e. the relationship between words based on the distributional hypothesis). Therefore, it is not a case of replacement, but of composing these two translation strategies. As a result, if a source is generated with syntactic errors, the generator would produce an improved source by using the error messages as feedback.

In What is it like to program with artificial intelligence?, the authors cast their net broader and try on a few different metaphors to see how they might fit LLMs.

• LLM as a pair programmer: Copilot’s original pitch. Sometimes you drive, sometimes you navigate. It’s your supercharged rubber duck.
• LLM as search: Skip Google and StackOverflow, ask ChatGPT/Copilot.
• LLM as compiler (there it comes up again!): the march of programming to higher levels of abstraction and being declarative continues unabated, only to end up in… English?

The authors point out scenarios where each of the above metaphors applies, and where it breaks down. Because none of them is a perfect fit, they say that programming with LLMs is an entirely new, distinct way of programming. It certainly feels that way.

Hoping to make videos soon covering these in a bit more detail.

I wrote up some of my initial thinking on this topic in a blog post titled LLMs are compilers, which should tip you to which side of the debate I’m on. I don’t think this genie (or Pandora, depending on how you look at it) is going back into the box, so now it’s mostly a question of how best to use it as a tool.

OK, let’s go:

GitHub Copilot is a technology that is designed to help you write code, kind of like your partner in pair programming. But, you know, it’s not an actual person. It’s “A.I.”–whatever that means.

In principle this sounds like it might actually be a good thing right? I know several people, who I respect, that use it as part of their daily work. I’ve heard smart people say AI coding assistants like Copilot will democratize programming, by making it possible for more people to write code, and automate the drudgery out of their lives. I’m not convinced.

Full disclosure: I’ve been a paying customer of Copilot for a few months and have been using it for my own miscellaneous scripts, mostly Python and Emacs Lisp. (Note with big blinking red lights: this is for personal stuff only. Don’t use Copilot or products like it inside your company unless they have an explicit policy allowing it!) Speaking only for myself, I get orders of magnitude more value from it than the $10/month it costs. How? Copilot (often used in conjunction with GPT, and now, Bard) has reduced the activation energy for the small scripty projects I want to undertake from being just slightly over a weekend (which meant they would get abandoned or not even started) to fitting comfortably within a weekend. And that has made a world of difference, at least for me. I suspect the same story will play out across programmers in general, where it will serve to lower the friction and frustration of programming just enough to increase their overall output.

Here’s why I don’t use Copilot (or ChatGPT) to write code:

Copilot’s suggestions are based on a corpus of open source code in Github, but the suggestions do not mention where the code came from, and what the license is. GitHub is stealing and selling intellectual property.

This is the strongest argument against Copilot–that they’re license-laundering code. This is also the reason that most (all the ones I know of, certainly) big or medium-tech companies outright ban the use of Copilot and similar products for in-corp use. The IP risks are simply not worth it. Until the legal aspect of Copilot is tested in court and some clear precedent is established, that is not likely to change.

But there might be a chicken-and-egg problem here: large corporations that produce code are steering clear of using Copilot, so there is unlikely to be a major lawsuit that would test this in courts. The biggest of big-tech (FANG companies) have internal codebases large enough that they can train LLMs purely on their own codebase to offer completions to their own engineers in-corp, and never worry about getting sued for license infringement. Maybe the best chance of a test case reaching the courts is a small startup using Copilot to develop a product that becomes a huge hit, at which point it would be large enough (for someone) to sue.

Copilot lets you write code faster. I don’t think more code is a good thing. The more code there is, the more code there is to maintain. Minimalism in features is usually a good thing too. Less really is more.

This argument has the same shape as Charles Petzold’s classic Does Visual Studio rot the mind?, which was written back in 2005 and lamented the rapid onset of IntelliSense as a basic tool of programmers everywhere. I am sympathetic to this argument, but I’m not sure this is going to persuade programmers who are in a rush to put a product out into the market. If a tool let’s them ship 1% faster, they will use it.

There is a whole different argument around what type of coding needs IntelliSense (and now Copilot) vs what can be done bareback, but the “minimalism is beautiful” argument depends on your dependencies being minimal and beautiful. When you’re using a library with thousands of classes, each with hundreds of methods, that argument falls apart.

As more and more programmers use Copilot it creates conservativism in languages and frameworks that prevents people from creating and learning new ways of doing things. Collectively, we get even more stuck in our ways and biases. Some of the biases encoded into LLMs are things that we are actively trying to change.

Another strong argument. Copilot sets up a positive feedback loop that greatly reinforces current languages, libraries and frameworks. Doing ML-heavy work in Python? Copilot is so stunningly accurate in that domain (no doubt due to the mountains of code that use pandas and scikit and pytorch and tensorflow) that if a newer, better ML library comes along, one of it’s greatest barriers to adoption will be programmers reticent to use it because Copilot doesn’t offer good completions for it yet. Same goes for new languages. A chicken-and-egg problem again.

I predict that in order to get traction on new libraries or languages their developers will have to build additional tooling that can somehow “inject” good completions for that particular new entrant.

Developers become dependent on Copilot for intellectual work. Actually, maybe addicted is a better word here. The same could be (and was) said about the effect of search engines on software development work (e.g. Googling error messages). But the difference is search results need to be interpreted, and the resulting web pages have important context that you often need to understand. This is work that Copilot optimizes away and truncates our knowledge in the process.

The comparison with addiction is one that Petzold makes in the above-linked piece. The argument here is that Copilot is letting programmers get away with a shallower understanding and lesser knowledge. But the exact same argument can be made of every advance that has continously raised the level of abstraction of programming over the decades. Assembly language obviated the need to understand binary programming. Compilers obviated the need to understand assembly language. We’re at the point where one could be perfectly adept at accomplishing tasks in Python without actually understanding how a CPU works. The march of progress continues.

Will there need to be an “elite” class of programmers that build all this infrastructure? Someone has to understand the full stack enough to build OS kernels, compilers, language runtimes, garbage collectors and ML libraries. We’ll be in trouble if that class dwindles too much. C is not hip, it’s old and tired, but every major OS kernel is still written in C. For the millions of programmers that pick up Python with YouTube tutorials (and that is a wonderful thing!), we still need a few dozen to get drawn into the Linux kernel source and get fascinated by C and pointer chasing, to preserve our ability to manufacture the concrete used to build our digital metropolis. Hopefully things like Copilot lower the barrier enough on the Python end of the pipeline that we still get the trickle we need on the C end.

Copilot costs money. It doesn’t cost tons of money (for a professional person in the USA) but it could be significant for some. Who does it privilege? Also, it could change (see point 4). Remember who owns this thing.

True. $10/month when translated to most places in Asia would be significant enough to keep this out of the reach of most young programmers. With the advent of open-source LLMs that continue to get more capable, hopefully we’ll see this price drop significantly, or even soon reach the point that a fairly capable model can be run entirely locally.

How much energy does it take to run Copilot as millions of developers outsource their intellectual work to its LLM infrastructure? Is this massive centralization and enclosure really progress in computing? Or is it a step backwards as we try to reduce our energy use as a species?

I’m an unabashed accelerationist, so I’d argue this is besides the point. But even if you buy into the premise that we need to reduce our energy consumption, one could argue that centralization leads to efficiencies of scale, and for all we know it might actually be more energy-efficient to consolidate computation happening on millions of clients and bring them into a datacenter.

What does Copilot see of the code in your editor? Does it use your code as context for the prompt? What does it store, and remember, and give to others? Somebody has probably looked into this, but if they have it is always up for revision. Just out of principle I don’t want my editor sending my code somewhere else without me intentionally doing it.

Very fair point and I completely agree. More transparency would go a long way towards more developers becoming comfortable with technology like this. A local log of everything sent to the server would be great.

Working with others who use Google Copilot makes my job harder, since they sometimes don’t really understand the details of why the code is written a particular way. Over time Copilot code can mix idioms, styles and approaches, in ways that the developer doesn’t really understand or even recognize. This makes maintenance harder.

This depends on how things like Copilot embed into the overall development flow. Do you have linters or static checkers enforcing basic formatting and style? Do you have a local style or best practices guide? Do you have a robust code review culture? Yes, if you never look at completions from Copilot with a critical eye and accept blindly everything it suggests, you’re likely to end up with a crappy codebase. But you will also end up with a crappy codebase if you take the first draft of your code and check it in without any code review.

As far as I can tell the only redeeming qualities of Copilot are:

Copilot encourages you to articulate and describe a problem as written prose before starting to write code. You don’t need Copilot for this. Maybe keep a work journal or write a design document? Maybe use your issue tracker? Use text to communicate with other people.

Copilot is more interactive than a rubber duck. But, it turns out Actual People are even more interactive and surprising. Reach out to other professionals and make some friends. Go to workshops and conferences.

Copilot makes me think critically about machine learning technology, my profession and its place in the world.

Could not agree more! Sharpening your writing skills, discussing problems with colleagues and looking critically at the new shiny thing— these are all things every programmer should be doing.

Using not just Copilot but an LLM as a “smart” rubber duck that you converse with can be an intellectually stimulating and effective way to iterate towards a solution, and even get a deeper understanding of the tradeoffs involved. A couple of great examples of this mode of programming were written up by Prof. Crista Lopes in a series of blog posts.

Maybe my thinking on this will change. But I doubt it. I’m on the older side for a software developer, and (hopefully) will retire some day. Maybe people like me are on the way out, and writing code with Copilot and ChatGPT is the future. I really hope not.

But some good news: you can still uninstall it–from your computer, and from your life.

I share some of Summers’ discombobulation. The day after GPT-4 was demo’ed I wrote:

A couple of years ago I thought I had a decent grasp of the trajectory of programming as a field, and as a career path. Languages, frameworks and libraries would get better at modeling and abstracting more complex programs, safety vs performance would be less and less of a conflicting tradeoff, compilers would get better, IDEs and tools would get better, and underneath it all, even if Moore’s Law was hitting a wall, we’d just continue throwing more cores at larger problems. But at the end of the day, the immense cognitive leverage that comes from encoding something into executable software would ensure that programmers were always in short supply. But now I feel unsure and disoriented, not certain about where I stand.

Potters make art, but they must love the physical feeling of clay. Code is our clay, and we don’t know what it’ll feel like in a year or two.

The AI doomers are everywhere, and they want to start blowing up stuff, and I want to talk about it.

When Yudkowsky blithely advocates airstriking datacenters that could possibly be training an AGI, even at the risk of that setting of a nuclear exchange, that is the point at which the podcast hosts should’ve gone – woah woah woah, slow down, that’s kind of batshit crazy. You’re willing to actually doom humanity for real before it’s theoretically doomed at some point in the future by AGI. I know you’re constructing some pretty sophisticated arguments that you’re totally convinced of, but I’ll withold my unabashed support for just a little bit, if that’s ok with you.

There’s obviously been a rigorous debate about the merits of the AI doomer argument, but I want to highlight some considerations that have not even entered into the debate, much to my surprise.

Long-termism

But a bit of background first. The moral-philosophical underpinnings from which this line of thought comes makes this stance consistent within that world view.

There is a passage in the Bostrom Superintelligence book, which is the bible of the AI doomer movement, that explains the basis of the philosophy of long-termism (who could even argue with a name like that?).

The total “endowment” – the thing we have the highest moral duty to protect – is the total number of potential human minds, both real (in the flesh?) and virtual, that could live on a substantial fraction of the habitable planets (and even structures in space that are not planets) up to the space reachable by light from Earth, up to the time of the heat death of the universe. That’s estimated to be 10^58 minds.

The AI doomer camp was birthed out of the philosophy of long-termism. Bostrom was one of the founding philosophers. Yes, the same Bostrom who wrote Superintelligence. Many others have crafted excellent critiques of that philosophy, so go read those.

But the gist of it is that it is being used to justify some pretty horrendous suffering wrought upon the real humans beings that are alive today, in order to make sure that the 10^58 humans in our potential future have a chance at existing. What’s 10^9 sacrifices when compared to saving 10^58 ? Not even significant.

And that is the reason why the Bostrom-Yudkowsky camp are so comfortable proposing inflicting massive suffering today to reduce that probability of existential wipeout by even a tiny bit. It is guilt-laundering from the present into a future so vast that it is absolved by dilution.

Police state panopticon

OK, let’s even step over the “it’s better to provoke nuclear war than get to AGI” argument. Let’s look at the kind of moratorium they are calling for, and the enforcement that requires. Think about what kind of precedent this would set. Forget for a minute the specific field of AI that the AI doomers are talking about.

What they’re arguing for is a global government enforcement mechanism that would subject both private corporations and citizens to an all-seeing police state. Because that’s what it would take to peek into every data center everywhere. That’s what it would take to ensure that GPU manufacturers build tamper-proof GPUs that call home and report malicious training loads, for some definition of malicious that the doomer cabal deems acceptable. And of course, it won’t take only monitoring GPUs, because one could train an AGI on CPUs. So we’re already at a place where we’d need to police all computation, whether it happens on GPUs or CPUs. No more encryption. Every computational load anywhere will be subject to random, or ongoing, surveillance to ensure that the humanity-destroying AGI isn’t being trained on it.

Keep going down the compute chain. True enforcement would require monitoring and watchdogs at every semiconductor fab. And to prevent the possibility of someone maybe spinning up their own little fab hidden from the government panopticon, you would give this police state even deeper powers to monitor every aspect of life, industry and consumption. After all, you’d want to detect large draws of electricity in case it was being used to train an AGI.

And this politburo once established will of course want to save humanity from other new technologies that pose existential risks. So we’ve gone and created a body outside democratic and judicial guardrails that has the power to squint real hard and declare some class of human endeavor dangerous enough to pose an existential risk, then put a stop to it.

That is the path the AI doomers want to send us down. And this question of individual liberties and preserving free societies never comes up. Not once in the Superintelligence book, and not once in the blitzkrieg of podcast appearances that Yudkowsky has gone on recently. None of the hosts thought to ask about it either. I thought that was a large enough omission to make this rant video.

History

History is not devoid of humanity successfully dealing with what were at the time thought to be existential risks.

Example #1 is nuclear power. We made atomic bombs, and then the entire world got together to make sure that we didn’t wipe ourselves out while preserving the ability to go to war with each other. So, yaay, score for humanity!

Another example is the Asilomar Conference. Back in 1975 the prominent biotechnologists of the day got together to discuss how to safely perform research that involved recombinant DNA. They were worried that splicing DNA around would create a killer virus or bacteria or cancer cell that would spread out in the world. They came up with restraints on themselves, and principles for how to safely carry out research while moving the field forward. And all this without dragging government and regulators into it.

So scientists can self-organize to deal with risks, they can self-regulate, they can find ways to push the boundary of knowledge further without shutting down the whole endeavor, and certainly without advocating for airstriking labs.

Closing

The AI doomers have all the attention right now. Perhaps because we’re all worried that AI will take our jobs and strip us of our humanity. That voice saying “AI will kill us all! Shut it down now!” sounds pretty appealing. But it deserves a bit more pushback.

The first paper was The effect of CAD on Architecture Students’ Creativity and Enthusiasm. It looked at self-reported answers from architecture students.

The study found that architecture students do not perceive CAD applications as having potential to make them lazy in design. Furthermore, student’s ability to develop conceptual patterns, creative forms and novel ideas was found to be enhanced by the use of CAD. Most of the respondents (77.8%) agree that CAD enhances creativity. Lack of design drive and enthusiasm flourish among the undergraduate students because the requirement to present design solutions in manual drafting is perceived as burdensome and a waste of time.

The second piece was more an essay and analysis: CAD and creativity: Does the computer really help?

When CAD helps creativity

• Students who have great ideas but are not good draftsmen and drawers use CAD to express ideas they would not otherwise have been able to
• Even experienced architects say that CAD allows them to push the boundaries of how complex or intricate their designs are
• Simple CAD systems can be used by non-professionals to design buildings

Can CAD hinder creativity?

• Many top architects still work with paper and physical forms, and see benefits of CAD to be not large enough to justify the investment in learning it
• Anecdotal trend among students: they become proficient in the software, but their creations lack “real” creativity: “They may look extremely convincing, they might be original, but they are most certainly not good design.”
• On virtuosity without soul, the ease of blinding with polished presentation that lacks deeper substance:
  • “Theoretically, it has always been possible to find excellent presentation combined with poor design. However, before the advent of CAD, it seldom happened in practice. This is probably because the visual sensitivity needed to design and to draw well are so similar that it would be unlikely for a student to be skilled in one area but not in the other.”
  • “So it is possible to put forward computer presentations that look attractive and even dazzling, that seem authoritative, while the architecture so represented is really quite awful.”
• They draw a parallel with desktop publishing: “This phenomenon can also be illustrated with reference to the graphic design that proliferated with the advent of the Apple Macintosh computer. We started to see documents obviously produced by people who owned a Mac, with its (at the time) revolutionary what-you-see-is-what-you-get interface and multiple fonts. These documents characteristically contained as many fonts as possible, apparently merely in order to exploit this new and wonderful facility.”
• Computer makes it easy to do “hard” things without understanding underlying principles and constraints: “Even worse, some of these forms are relatively easy to generate in CAD but are hard to represent in manual perspective—for example, shell forms based on ellipsoidal sections, rotations of curved parabolic forms and so on. Perhaps this encourages students to believe that because they have drawn something infrequently seen, they are being creative.”
• CAD does not understand or represent the materiality of the structures being designed. The shapes are abstract.

Of course, if you’re a programmer, you will note the obvious parallels to the debates around IDEs, and lately, Copilot and LLMs.

I’ve always been fascinated by the question of how we as programmers relate to our tools, how that makes us feel about what it is we actually do, and about our identity. Are we craftsmen? Are we factory workers?

It’s a question I’ve been contemplating for more than a decade on this blog. Some prior writing:

• Operation vs expression
• Sharp tools, dull minds
• You can’t run with the machines
• De-skilling
• Automating software engineering
• The perils of being a knowledge worker

I’m happily using GitHub Copilot for my personal code. It has made me churn out more Emacs Lisp and Python, and the prospect of actually getting a little project done in a few precious spare hours on the weekend has definitely egged me on and made me more enthusiastic about experimenting with and starting new things. But what does all this mean for us as programmers?

The release of GPT-4 yesterday and the subsequent demo has made me feel a mix of excitement, melancholy and disorientation.

A couple of years ago I thought I had a decent grasp of the trajectory of programming as a field, and as a career path. Languages, frameworks and libraries would get better at modeling and abstracting more complex programs, safety vs performance would be less and less of a conflicting tradeoff, compilers would get better, IDEs and tools would get better, and underneath it all, even if Moore’s Law was hitting a wall, we’d just continue throwing more cores at larger problems. But at the end of the day, the immense cognitive leverage that comes from encoding something into executable software would ensure that programmers were always in short supply. But now I feel unsure and disoriented, not certain about where I stand.

I’m beginning to get more and more convinced of Matt Welsh’s take on how LLMs spell the end of programming as we know it (YT video of his talk). The last few decades saw steady linear improvements in programming languages and tooling. The next few years (who knows, maybe months?) are going to be uncharted territory, a step change. Like the saying goes – may you live in interesting times.

The hottest new programming language is English

— Andrej Karpathy (@karpathy) January 24, 2023

Programming in assembly language used to be a skill. That skill became irrelevant when good optimizing compilers could translate high-level languages to performant assembly (and when whatever tiny difference remained got swamped by gains in CPU speed). Nobody argues today that programming should be done in assembly.

Today we’re living through a similar transition to a higher level of abstraction. Code in HLLs is becoming the output of an even higher-level language– the natural language prompt to an LLM.

Here are a few examples to drive the point home:

Copilot easily passes “Intro to CS” courses: In this paper, the authors found that on a set of CS101-level programming problems, Copilot solved half of them right away (just feeding it the problem description itself), and solved 60% of the rest with only changes to the problem description.

Other observations have shown that when the output from tools like Copilot is wrong, developers avoid debugging the generated code, and either fiddle with the prompt until it works, or rewrite the code from scratch.

And here is Prof. Crista Lopes (1, 2, 3, 4) trying to grab the CS education community by the shoulders and shake them, after getting surprisingly good results with ChatGPT when implementing a lexer and parser for a toy programming language– a problem common in graduate-level compilers courses.

The neural network was able to understand the concrete symbolic reasoning of a Lox tokenizer purely by examples; it was able to generalize well beyond the examples; it was able to correct my specification mistakes; … bottom line: it is able to tokenize Lox programs without a single line of code being written.

Obviously, I am “programming” it by teaching it the rules using a combination of natural language and examples. But this is not programming as we know it. This is a blend of old ideals such as “do what I mean” and programming by example, but on steroids. While I am very familiar with these old ideals, I never thought I would live to see the day where they were a reality!

Taking the compiler analogy even further are systems like Parsel, which takes in a spec that mimics the way human programmers think, in that the spec decomposes the problem to be solved into smaller sub-problems, along with function signatures and a few input-output examples (i.e. unit tests), and produces code for that higher-level problem. The authors have recently made advances that let you omit the input-output examples, and even omit the function signatures, generating code from a natural language breakdown of the problem!

Parsel🐍 update: you now don't need to write function names/args at all - it's literally just indented natural language!

Images show 4 lines of Parsel and the generated 114-line Python code
Try or contribute here! https://t.co/vZ33rNFzuX
Last thread: https://t.co/SAJzbD9lKL pic.twitter.com/LsASh6OBhI

— Eric Zelikman (@ericzelikman) February 14, 2023

Of course, Copilot is not perfect. It still generates incorrect code, even though the hit-rate is pretty good. The question is: do you think it’ll get better or worse in the future? Programmers today never have to peel back the curtain to debug the assembly language generated by modern compilers (unless you’re a compiler writer). Sooner or later, code LLMs will achieve the same level of reliability, truly making them compilers from English to working code.

His rationale for that claim:

The key part of Emacs is that it’s mostly written in Lisp … There’s a core implementation that can read the file, put bits on the screen, and manage memory and buffers. Then, what makes it an editor full of features is all the lisp packages and, of course, the design of how the lisp packages interact with each other and with that base layer of the core immutable engine.

… almost everything in that core engine in Emacs’ case, can still be overridden or replaced. And so, VS Code has a similar architecture, where there is like a base engine … and it has a sort of a package manager and a whole series of interfaces for packages and an additional series of conventions for how packages should interact with the lower layers and with each other.

…. And so, the sort of extreme extensibility and the package ecosystem that you see in VS Code is a mirror of very similar architectural features in Emacs.

I agree that VS Code has a similar architecture to Emacs, in that it consists of a core language runtime, with most functionality built on top of that. For Emacs, that is Lisp. For Visual Code, JavaScript. But the Zen of Emacs (like the Zen of Python), is very, very different.

If I were to distill Emacs down to its two core ideas, they would be: barrier-less extension, and everything is a buffer.

Barrier-less extension: The way you change behavior in Emacs is by writing and evaluating Emacs Lisp expressions. You can do this anywhere in Emacs, which is to say, in any buffer in Emacs. Write elisp. Mark it. M-x eval-region. And now your editor is in a new universe. If you like that universe, you save that little elisp snippet in a designated place (init.el or something similar) so that you don’t lose it and Emacs evaluates it when it starts up again. This is ingrained into every longtime Emacs user. It is probably the second thing a newbie Emacser learns after the basic key bindings. This mode of changing the editor immediately by evaluating Lisp expressions is central to Emacs’ customizability and power.

Everything is a buffer: Emacs turns its full power towards fine-grained surgery on text. So why keep any part of it outside the reach of that power? The buffer as a central organizing concept is both simple and powerful. Every piece of text everywhere is in a buffer. It is subject to the same text manipulation. This includes shells and Dired.

When you combine the above two simple principles, you get nearly unbounded combinatorial power. Every shell interaction and its output becomes a text buffer. A directory listing in Dired can be edited (and then those changes persisted back!) just like a regular Emacs buffer. You can start climbing the levels of Emacs proficiency, a ladder that has no end. Because of this, Emacs can be the editor of a lifetime. You mold it to your usage over time. It becomes your tool forge.

VS Code lacks both of these ideas. But clearly, judging by its popularity and mindshare it has found success by doubling down on its strengths. It has just picked a different set of values than Emacs.

VS Code is a product from a giant tech company: Microsoft. One with a history of focus on developers, because they realized early how crucial developer mindshare was to maintaining the appeal of Windows. So VS Code has what Emacs doesn’t– product managers. People who create, articulate and then execute a strategy to capture a given market.

Emacs, by comparison, is the rag-tag Rebel Alliance. Maybe it also suffers from the Lisp Curse: Lisp is so powerful that problems which are technical issues in other programming languages are social issues in Lisp. That’s why instead of one canonical, well-maintained, bug-free implementation for a set of functionality (e.g. completion!!), it has twenty, with five of those actually solid and reliable enough for regular use, but with very differing philosphies and feature sets. Some might view that as a strength. Others may be turned off by it and turn to VS Code because It Just Works.

Speaking for myself, I’ve had to begrudgingly use VS Code once in a while simply because it is so prevalent and becoming the canonical editor (with all the tooling support that goes along with it) in more and more scenarios. But the paucity of its abstractions grates on me at every turn. The command pallette is a shadow of M-x in Emacs. There are no macros. The terminal doesn’t act like any text “buffer”. To “extend” it you have to write an extension, and there’s a ton of process and setup around it. You can’t just write JS expressions in a file and evaluate them to change the behavior of the editor. And on and on.

So no, I don’t think VS Code is the spiritual successor to Emacs.

Just a few days ago, the folks over at Meta AI released a paper that takes that much further, by incorporating external API calls into the training and inference steps of the LLM itself. Their system is called ToolFormer, and here’s a short video on it:

Their topline result is that when augmented with APIs in this way, smaller models like GPT-J (6B parameters) can outperform models like GPT-3 (175B parameters), that are an order of magnitude larger, on a variety of tasks.

But this paper argues that trying to get an LLM to do calculations is a misunderstanding of its strengths. LLMs are meant for understanding language, not calculation. The key idea is to separate the problem into two parts: understanding, and computation (or solving). Let the LLM do what it is good at– understanding– and then offload the computation to an actual programming language.

Check out below the quick video summary I posted last week. Also see their GitHub for the full source prompts they use.

Let’s see a few examples of how that works.

But first, an example of where LLMs fail (all examples in this post come from the text-davinci model. The LLM’s answers are demarcated with 🤖💬 and 🤖🛑).

Question: Roger has 5 tennis balls. He buys 2 more cans of tennis
balls. Each can has 3 tennis balls. How many tennis balls does he have
now?

Answer: Roger started with 5 tennis balls. 2 cans of 3 tennis balls
each is 6 tennis balls. 5 + 6. The answer is 11.

Question: The bakers at the Beverly Hills Bakery baked 200 loaves of
bread on Monday morning. They sold 93 loaves in the morning and 39
loaves in the afternoon. A grocery store returned 6 unsold loaves. How
many loaves of bread did they have left?

Answer:
🤖💬
The bakers started with 200 loaves of bread. They sold 93 loaves in
the morning and 39 loaves in the afternoon. 6 loaves were
returned. 200 - (93 + 39 + 6) = 62 loaves of bread left.
🤖🛑

Instead, we prompt it to not compute the answer, but to provide Python code to compute the answer.

Question: Roger has 5 tennis balls. He buys 2 more cans of tennis
balls. Each can has 3 tennis balls. How many tennis balls does he have
now?

Solution in Python:
def solution():
  """Roger has 5 tennis balls. He buys 2 more cans of tennis
balls. Each can has 3 tennis balls. How many tennis balls does he have
now?"""
  # Roger started with 5 tennis balls.
  tennis_balls = 5
  # 2 cans of 3 tennis balls each is
  bought_balls = 2 * 3
  # tennis balls. The answer is
  answer = tennis_balls + bought_balls
  return answer

Q: The bakers at the Beverly Hills Bakery baked 200 loaves of bread on
Monday morning. They sold 93 loaves in the morning and 39 loaves in
the afternoon. A grocery store returned 6 unsold loaves. How many
loaves of bread did they have left?

Solution in Python:
🤖💬
def solution():
  """The bakers at the Beverly Hills Bakery baked 200 loaves of bread
on Monday morning. They sold 93 loaves in the morning and 39 loaves in
the afternoon. A grocery store returned 6 unsold loaves. How many
loaves of bread did they have left?"""
  # The bakers started with 200 loaves of bread.
  bread = 200
  # They sold 93 loaves in the morning and 39 loaves in the afternoon.
  sold_bread = 93 + 39
  # A grocery store returned 6 unsold loaves.
  returned_bread = 6
  # The answer is
  answer = bread - sold_bread + returned_bread
  return answer
🤖🛑

You can check that running the Python code gives the correct answer (74).

You can do the same thing with logical problems, not just arithmetic:

Question: say "java" twice and "data" once, and then repeat all of
this three times.

Answer: java java data java java data java java data


Question: Repeat the word duck four times, but halfway through also
say quack

Answer:
🤖💬
duck quack duck duck quack duck
🤖🛑

OK it failed that. Let’s try the same with code-style prompts.

Question: say java twice and data once, and then repeat all of this
three times.

# solution using Python:

def solution():
    """Question: say java twice and data once, and then repeat all of
    this three times."""
    result = []
    tmp = ["java", "java", "data"]
    for i in range(3):
        result.extend(tmp)
    return " ".join(result)


Question: Repeat the word duck four times, but halfway through also
say quack


# solution using Python:
🤖💬
def solution():
    """Question: Repeat the word duck four times, but halfway through
    also say quack"""
    result = []
    for i in range(4):
        if i == 2:
            result.append("quack")
        result.append("duck")
    return " ".join(result)
🤖🛑

tl;dr– use LLMs for what they were meant to do– understand and predict natural language– and then leverage that for computation.

The goal of these hacks is to construct a computing setup that is inexpensive, upgradable and repairable. This makes them tinkerable. It’s easier (and more fun!) to play around and risk breaking old cheap hardware than that latest $2k shiny laptop. Computing and storage have made such vast leaps that for everyone but a tiny fraction of users anything made within the last decade is more than enough to be their daily computing driver. This is not aimed at those slinging around 4k video and needing color-accurate displays. It is for web browsing (isn’t every computer basically a Chromebook with some small extras at this point?), watching videos, and run-of-the-mill productivity stuff like word processing and spreadsheets.

Laptops

Old Thinkpads FTW, ‘nuff said.

They’re built like tanks so you don’t have to worry about dropping them once in a while. Back lid pops off easy so you can upgrade/replace RAM and disk (and even the CPU on most of them). The Thinkpad keyboard is miles ahead of any other laptop keyboard, even the latest ones. There are tons of videos on YouTube walking through how to upgrade every model out there. People have even swapped out the display and trackpad to match their liking.

You can find old Thinkpads in good condition for low 100s of dollars on eBay. If you go deep into the Old Thinkpad rabbit hole, you can even find plenty of non-functioning ones on eBay that are ~10s of dollars that can be scavenged for parts.

Some notable YouTube channels that focus on old Thinkpads (comparing models, reviewing them, upgrading them): Laptop Retrospective, Sebi’s Random Tech.

Desktop

I recently discovered the 1-litre (ultra-small) form factor via ServeTheHome’s Project TinyMiniMicro. They’re pretty much desktop machines, but in a much smaller form factor, while still retaining upgradability.

I haven’t looked at every brand in detail, but from a quick comparison it looks like Lenovo carried some of their Thinkpad philosophy into their Tiny line, in that they’re easier to open up and upgrade than the HPs and Dells. I recently bought an M710Q Tiny second-hand, and it was super-easy to add RAM and SSD. You can get in with one screw, and the SSD and RAM slots are tool-less, so no more messing with screws in there.

You can find a pretty decent used Tiny/Mini/Micro for around $200-$300 on eBay, and for that much you’ll typicall get a 6th or 7th gen Intel Core processor, 8 GB RAM and around 200GB SSD. Not all models have WiFi though, so watch out for that.

Or you could just dock your old Thinkpad.

Displays

I personally find that the element of my computing setup that most affects my productivity is the display. More specifically, having a large display. Turns out there’s empirical data to support that. There’s a tradeoff there between screen real estate and ergonomics so that you don’t end craning your neck from one side of the display to the other. I’ve found that at typical desk viewing distances (~3ft), 40-43 inches works pretty well. This will be different for everyone so you should experiment and find what works best for you.

But the hack is that 4k TVs (not computer monitors) of size 39-43 inches are around $200 these days, which is less than half of what a comparable monitor goes for. Around Black Friday you can easily find deals on the former for as low as $150. Yes, you can probably tell the difference if you pixel peep, but you’ll have to squint pretty hard.

Note that the default settings on these are optimized for TV viewing, not use as a computer monitor. Typically, you’ll need to turn down the saturation to something in the middle of the range, and the sharpness all the way down. Some of the recent makes and models seem to have caught on to the fact that they’ll get used as monitors and have settings bundled into “modes” for TV, PC and gaming.

May your hardware dollars go far and bring you joy!

Massive props to Donald Feury who created a script that combines ffmpeg and MoviePy to do just that.

I’ve been using that to edit the last couple of videos I created, and also been making minor improvements to it:

• the original script depends on the “silencedetect” filter in ffmpeg, which in my experience can sometimes leave long stretches of silence undetected, forcing me to edit the video manually and then render it again
• the original script uses a two-step process: first run the ffmpeg silencedetect filter, then use the output of that with MoviePy to render the final video

I’ve addressed those issues in this script which does the silence-detection using MoviePy’s audio APIs, eliminating the dependency on ffmpeg, while also making the script self-contained such that it does the job in one invocation. It has proven to be much more reliable at detecting all silent periods in the original video.

So I recently got a Thinkpad Trackpoint II keyboard, which I like to use with my MacBook when docked to an external monitor. Yes, I realize that’s mixed up.

But it happens to have a “PrtSc” key right on the space-bar row that is perfectly placed to be the “Cmd” key. Mac OS X, in its Keyboard Preferences, lets you remap some standard keys (Caps Lock to Ctrl, for example), but not PrtSc.

Following this documentation, this is the command I ended up with to map PrtSc to Cmd:

hidutil property --set '{"UserKeyMapping":[{"HIDKeyboardModifierMappingSrc":0x700000046,"HIDKeyboardModifierMappingDst":0x7000000E7}]}'

The keycode for PrtSc is 0x46, and for “Right GUI” aka Cmd is 0xE7, and you have to OR them into 0x700000000.

I’m using the name “Unix” to include the entire family of operating systems descended from it, or that have been heavily influenced by it. That includes Linux, SunOS, Solaris, BSD, Mac OS X, and many, many others.

Both major mobile OSs, Android and iOS, have Unix roots. Their billions of users dwarf those using clunky things like laptops and desktops, but even there, Windows is only the non-Unix viable OS. Almost everything running server-side in giant datacenters is Linux.

How did Unix win?

• It was built by programmers, for programmers. If you read the early papers describing Unix, you will see how the key abstractions (hierarchical filesystems, permissions, processes, interactive shells, pipes) have lasted conceptually unsullied for decades. That could only have happened if it exerted such a force field over geek minds that they propagated it.

• The majority of it was written not in assembly language, but in C, a higher-level language, hence making it portable. This enabled relatively easy ports to a wide variety of hardware.

• A freak historical accident. AT&T Labs, where it was developed, was forbidden under its anti-trust settlement from commercializing products unrelated to its core telecom business. Hence, Unix was licensed very cheaply to universities, including UC Berkeley, which subsequently built one of the more influential branches of the Unix family tree—BSD. Apparently, AT&T classified Unix as industrial waste for tax purposes when licensing it!

• Unix spread through academia, and those students spread it through corporations after they graduated. It’s a strategy that has been used by every major tech company, only in this case it was organic.

• And then came the Internet, and the whole universe of daemons, tools, protocols, and utilities that undergirded it was built natively on Unix. That wasn’t a huge surprise because a lot of the people that built the Net were Unix natives. BSD open-sourced its TCP/IP stack, kicking off its wide adoption outside the military.

• By the late 90s and early 2000s, Linux started taking over the server-side. It has become the ultimate virtuous cycle in open source. When picking a new kernel one is virtually forced to go with Linux because of the huge community and massive engineering that has gone into solidifying it. That’s probably why Android picked Linux even though it was running on the other end of the hardware spectrum.

So how and why would we move past Unix?

• Unix, which got standardized into the POSIX spec, has accreted a tremendous amount of complexity over the decades. The POSIX spec is 3000 pages long. Linux now has nearly 400 system calls. What started as a clean, pure, minimal and elegant set of system abstractions has become a complex beast.

• The very idea of the OS providing general-purpose abstractions with wide applicability is being challenged. When Unix was created, IO was orders of magnitude slower than CPU. There were enough CPU cycles to burn to provide these high-level abstractions, like a complex hierarchical filesystem. Now IO can easily saturate CPU. At least on the server side, folks just want to get the fastest performance out of their hardware. That’s leading to the rise of frameworks like SPDK and DPDK that bypass heavy OS abstractions for storage and networking in favor of applications directly accessing the raw hardware, and rolling whatever abstraction they do need on their own.

• The entire ecosystem in which an OS exists is changing. Application-level experiences are hermetic and make the underlying OS more or less irrelevant. Just look at Android or iOS or ChromeOS. Programming to a virtual machine (like the Java VM) makes programmers much more invested in their PL/runtime than the OS.

• The above two points raise an interesting question: what does a modern operating system want? Look, for example, at Fuchsia, which is going in the direction of a microkernel with capabilities, that pushes most drivers and OS services out to userspace. These are ideas that have been floating in academic OS research for decades, but could never gain real mainstream acceptance because of the high barrier to making a viable real-world OS. This is the effect Rob Pike was talking about in his “Systems Research is Irrelevant” talk. But the prior two trends are finally dislodging the iron grip of Unix, and OSs are becoming interesting again!

I started putting out videos for the same reason I have a blog: I want to share knowledge, and learn from the feedback. I have learned so much from stuff other people have published, and I want to put a few drops back into that ocean.

Also, committing to a regular cadence of putting out content (once a week, every week) is a calming antidote to the general chaos of modern life.

Why YouTube? Why videos? Because I feel that’s where the river of attention is. And because it lets me experiment with a new medium. I believe Kevin Kelly when he says that screen literacy is a necessary modern skill, and should be taken seriously. I’m constantly amazed at what a fantastic place YouTube is for pedagogy, and the vast range of creators with their own unique voices and personalities putting out videos that can teach you everything under the sun.

As someone raised firmly within the walls-of-text tradition, it was very uncomfortable for me at first. But this was a different flavor of discomfort: this was unlike a fear a public speaking, which I don’t have any problems with. I can’t quite put my finger on it. My first video didn’t even have a voiceover. My later ones have my talking head in a rectangle in the corner. Maybe I’ll progress to the stage of being comfortable with me taking up the entire frame.

The majority of my videos so far have been part of my “Read a paper” series. I pick a technical paper, and walk through it, highlighting parts as I go along. So far they have all been computer science papers, with a strong bent towards old classics published in the late sixties and early-mid seventies, usually by Turing Award winners. I intend to branch out, both to more recent CS papers, as well as papers outside CS that I’ve found interesting and profound.

Why am I reading papers in videos? It was a format I instinctively gravitated towards. Only later could I shake an articulatable rationale out of my head. The thing is: for the longest time I’ve harbored deep misgivings about popular science and technology writing. I’ve come to the conclusion that if you really want to understand a new technical topic, you cannot rely on a pop-sci reading of it, and must bite the bullet and go read technical texts. And I realize that’s hard and seems intimidating. I want to play the part of someone holding your hand as you walk through a technical paper, beginning to end, without simplifications, with the original words of the author in front of you. I guess this is where my walls-of-text upbringing comes to the fore again: even though these are videos, my goal is to pull you into the text, and stress the primacy of it.

I do eventually want to make more opinionated videos (kind of like the blog), but I need to get more comfortable with this medium before I can do that. Sticking to the straight technical details makes it a bit easier for me right now.

I can’t get myself to beg for likes and subs on camera, but I’ll do it here: I hope you’ll join me on a whimsical journey into real technical details with real technical papers. Please subscribe!

Here’s what I’d like to see next in laptops: make the keyboard area a full-on touch screen. Something like the Microsoft Courier, but meant to be used in a laptop, typing-first form-factor rather than a vertical booklet form-factor.

This would get over the biggest UX complaint with the current generation of touchscreen laptops, which is that raising your arms to reach out and touch the screen is something that would tire your shoulders if you had to do it very often, and breaks from the rhythm of whatever you were typing on the keyboard.

You can project a full-touch keyboard on the screen. With some form of haptic feedback it could be made to feel halfway like a real one. But because it’s drawn on the screen, it can support all the modern features we’re used to from phone/tablet keyboards, like suggestions, swiping, and altering layout for special characters and emojis.

When not typing, you could use the entire bottom of the laptop as a touch pad. In general, for apps that use a lot of manipulation rather than typing (photo and video editing), you could get rid of the keyboard and use the entire bottom of the laptop as a gesture input area.

When learning keyboard shortcuts, they can be highlighted on the virtual keyboard.

And hopefully, you’ll never have to worry about spilling a drink on your laptop keyboard and ruining it!

People outside Silicon Valley may think of this book as entertaining and illuminating, but if you’re part of it (even if not geographically but within it’s extended mind sphere), this book will mostly read as real.

Every industry needs a metaphorical gaslamp to draw moths. That is the only way it can sustain itself from generation to generation. Actors have LA. Techies have Silicon Valley. Or more specifically, the Silicon Valley startup.

This book even taught me a lot about the ins and outs of the byzantine world of online advertising and targeting. The narrative of Twitter and early Facebook fumbling their way through trying to stay solvent by digging into various crevices of the online ads world is not just entertaining but also has some very pedagogically instructive passages.

The author makes a page turner out of his startup experience. He even makes a page turner out of his post-startup big-company experience. Because truth is stranger than fiction.

The biggest compliment I can pay the author is that he made me finish a book, something I hadn’t done in months. Kudos!

We could call this strange geography created by technology “AirSpace.” It’s the realm of coffee shops, bars, startup offices, and co-live / work spaces that share the same hallmarks everywhere you go: a profusion of symbols of comfort and quality, at least to a certain connoisseurial mindset. Minimalist furniture. Craft beer and avocado toast. Reclaimed wood. Industrial lighting. Cortados. Fast internet. The homogeneity of these spaces means that traveling between them is frictionless

• Airspace itself, and the underlying phenomena behind that aesthetic is much older than Airbnb and Foursquare. Airspace is the aesthetic of how the the western multi-national corporation wants to be seen, both overtly and subconsciously: powerful yet gentle, controlling yet subservient, sharp yet organic, but above all—spanning the globe and scaling to massive flows of products, materials and ideas. It is the aesthetic of the functional and comfortable office, every one as similar as possible to the home office, so an employee has to never spend any mental effort to adapt.

• I personally view airspace as a huge boon and am thankful for it. I am thankful that I can expect to get a non-fat vanilla latte in Delhi or Shanghai or Dubai that is guaranteed (by an immense and complex machinery of people, machines, processes and incentives) to taste exactly the same as the one that kickstarts my mornings in my Californian suburb. I am thankful that when I check into a big chain hotel in a city I’ve never been to I can expect a well-known and recognized room, amenities and level of service. Some may find this stultifying and stifling, may say that it eradicates the local flavor and imposes a sameness on everything and every place. But the local flavor and organic chaos is still out there, outside the hotels and the Starbucks and on the crazy streets, or in a national park far away from those, and I can go seek it if I wish.

• This homogeneous aesthetic extends not just to physical spaces, but also to lifestyles, life scripts and people’s desires. Go to any well-developed city with a strong middle class, and people there have mostly the same lifestyles (work, commute, weekend) and want the same material things (houses, cars, vacations). When was the last time you personally met someone whose lifestyle and desires were a huge surprise to you?

• Speaking for myself, the “aesthetic” that speaks to me is that of intimate and private spaces (even if they are in public view). Work desks with old food, empty soda cans, three-month old post-its with todo notes, and coffee circles. Sinks with dishes. The constant aftermath of a yard where kids play. This is the aesthetic of messes. And a mess is personal and intensely revealing. Every clean and designed space is a wall behind which to hide one’s true personality. But look at someone’s mess closely for a while and you feel like you’ve known them for ages.

Gist. Also pasted below.

/*jshint esversion: 6 */
'use strict';

// Church numerals in ES6.
// c.f. https://en.wikipedia.org/wiki/Church_encoding

// Zero is the identity function.
let zero = (f => x => x);

// Successor: apply function one more time.
let succ = (n => f => x => f(n(f)(x)));

let one = succ(zero);

// Convert a Church numeral into a concrete integer.
let plus1 = x => x + 1;
let church2int = n => n(plus1)(0);

// Convert a concrete integer into a church numeral.
function int2church(i) {
  if (i === 0) {
    return zero;
  } else {
    return succ(int2church(i - 1));
  }
}

// Add two Church numerals.
let add = m => n => f => x => n(f)(m(f)(x));

// Multiply two Church numerals.
let mult = m => n => f => x => n(m(f))(x);

// Exponentiation: n^m
let exp = m => n => n(m);

/////////////////// Some random operations.
console.log(church2int(int2church(10)));
console.log(church2int(zero));
console.log(church2int(one));

let c11 = int2church(11);
let c22 = int2church(22);
let plus_11_22 = add(c11)(c22);
console.log("11 + 22 = " + church2int(plus_11_22));

let mult_11_22 = mult(c11)(c22);
console.log("11 * 22 = " + church2int(mult_11_22));

let two = succ(one);
let ten = int2church(10);
let c_two_e_10 = exp(two)(ten);

console.log("2 ^^ 10 = " + church2int(c_two_e_10));

… evidence-based algorithms more accurately predict the future than do human forecasters. Yet, when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. This phenomenon, which we call algorithm aversion, is costly, and it is important to understand its causes. We show that people are especially averse to algorithmic forecasters after seeing them perform, even when they see them outperform a human forecaster. This is because people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake.

(emphasis mine)

As someone who makes a living from designing and implementing algorithms and presenting their results to people, this worries me (as it should any programmer!).

How can we beat this irrational bias against algorithms? The authors stop short:

Finally, our research has little to say about how best to reduce algorithm aversion among those who have seen the algorithm err. This is the next (and great) challenge for future research.

Note that demonstrating the superiority of an algorithm is not a solution, because people discount that and still prefer inferior results based on human judgment.

Some ideas off the top of my head:

• “Humanize” algorithms: highlight the people and teams that build them, and their dedication and hard work. Just like people associate “taxi” more with their human drivers rather than the mechanical cars they drive.
• Put humans between the algorithms and those consuming their output: Consider a core group of rationals who do not suffer from algorithm aversion, and share and forward the results of algorithms. This may not work for real-time lookups.
• Tie an algorithm’s fallibility to human error by the algorithm’s designer/implementer.

Any other ideas?

Some software writers worry that their profession’s push to ease the strain of thinking is taking a toll on their own skills. Programmers today often use… IDEs to aid them in composing code… they typically incorporate auto-complete, error-correction and debugging routines and… refactoring. But as the applications take over the work of coding, programmers lose opportunities to practice their craft and sharpen their challenge. “Modern IDEs are getting ‘helpful’ enough that at times I feel like an IDE operator rather than a programmer,” writes Vivek Haldar, a veteran software developer with Google. “The behavior all these tools encourage is not ‘think deeply about your code and write it carefully,’ but ‘just write a crappy first draft of your code, and then the tools will tell you not just what’s wrong with it, but also how to make it better.’” His verdict: “Sharp tools, dull minds.”

Thanks for the citation!

Lest that makes me sound like I’m advocating going back to not using IDEs, my position is a little more complicated than that. Later in that post:

Am I arguing for primitive tools? No. What I’ve described in the previous paragraph are my personal feelings while writing code. The tools (at least, the good ones) encode the knowledge and hard-won lessons of an entire army of programmers. They often point me to issues I’m blind to while writing code because I’m in such a rush to just make the damn thing work. In aggregate, they lead to a cleaner codebase.

So what am I saying then? I’m saying that we should let the tools help us without becoming crutches, to let us write sharp code without dulling our minds. That sounds paradoxical. It is a subtle mental stance one takes towards one’s work, tools, and output.

At some level the basic lesson of Carr’s book is the same: to not let automation make you complacent and loose the basic understanding of the thing being automated. It’s just hard to do.

Here’s an idea that sounds implausible today but might be practical in a decade or so: have pilots fly all commercial jets remotely from a “flight center” on the ground. There might be one such center for a designated geographical area. They might hand planes off to one another. Air Force pilots already fly drone missions halfway across the world remotely from a base in Nevada, so at least an early version of this idea is already being practiced.

That would mean pilots are fully engaged and active during their shift, rather than being lulled by hours of autopilot after a 2-minute manual takeoff. When something goes wrong on a flight, there will be a roomful of pilots to help out.

The major hurdle to implementing this will probably not be technical but getting people to board a commercial jet that doesn’t have a pilot on-board.

The issue with automating certain jobs is that their scope doesn’t expand. For example, a pilot performs certain actions, and if you automate 90% of those, the pilot is left with the same amount of time but only 10% of their prior workload. Contrast this with software development. Many tasks that were manual a decade ago are now handled by tools: automated build/test/deploy, IDEs that understand code and can perform sophisticated refactorings, infrastructure for rent etc. But today’s software developer still has their hands full because the overall complexity of developing software and the scope of problems that software tackles has grown too. Autocomplete in the IDE might have made them soft when it comes to remembering the APIs, but they do have bigger problems to grapple with.

A recent paper looks at a large cross-section of open source projects to analyze how adding people to projects affects their productivity.

They find that production intensity , as measured by number of commits, grows super-linearly with the number of project members. The median exponent was 4/3, which works out to a productivity boost of a factor of 2.5 when number of members is doubled.

Not surprisingly, the exponent was smaller (but still super-linear) for larger projects, but also showed less variability.

(For the link, I thank Nat Torkington’s Four Short Links. He somehow manages to come up with four interesting links every day.)

• Your understanding of the domain, or the domain itself, has changed drastically in the process of building the first version.
• The scale for which you built the initial system has grown by more than an order of magnitude. The right design for millions of users is very different than that for thousands.
• Newer languages and runtimes make the product much better along some dimension: deliver better performance, be more maintainable etc.
• You have accumulated so much technical debt that starting from scratch is easier than untangling the mess you have. This is often the most misleading reason, because engineers always prefer a system they wrote from scratch to inheriting one which works just fine. But even so, it does happen.

None of the above takes away from what Spolsky wrote. A rewrite is still a risky endeavor, even if one or more of the above conditions hold true, and should always be approached with caution and humility.

His argument was much stronger in the era of software that was shrink-wrapped to PCs, because the slow rate of change of the surrounding environment made many of the above reasons inapplicable.

But server-side software that powers online services is always in a constant state of flux. There is rarely any concept of “done”. The product might look more or less the same from the outside, while the backend code constantly churns to deal with growing scale, new stacks, and other services it depends on.

• on weekdays, you travel from home to work in the morning, and back in the evening; perhaps punctuated with a school or daycare drop-off and pick-up.
• on weekends, you either mostly stay at home or frequent a small set of haunts (beaches, restaurants etc).
• infrequently, you break out of this pattern by leaving the orbit of work-home to go on a vacation.

This is easy to figure out by observing someone for a short period of time. Of course, your phone already does this.

The above model of your life could be described in a few kilobytes. It is a zeroth-order machine learning problem. Is that depressing? Does that mean your whole life has a few kilobytes worth of complexity?

Free will can now be defined as the ability to surprise a prediction model.

— Vivek Haldar (@vivekhaldar) May 14, 2014

John Foreman, in his poignantly named piece “Data Privacy, Machine Learning and the Destruction of Mysterious Humanity”, takes a view of the field from the inside, and it sounds like that of a physicist working on the Manhattan Project worrying about the impact of what they’re building.

Our past data betrays our future actions, and rather than put us in a police state, corporations have realized that if they say just the right thing, we’ll put the chains on ourselves… Yet this loss of our internal selves to the control of another is the promise of AI in the hands of the private sector. In the hands of machine learning models, we become nothing more than a ball of probabilistic mechanisms to be manipulated with carefully designed inputs that lead to anticipated outputs… The promise of better machine learning is not to bring machines up to the level of humans but to bring humans down to the level of machines.

That is a dim, if plausible, view.

It is based on a pessimistic view of humans as always giving in to nudges to behave as our worst selves. I hope that we are better than that. We are already surrounded by advertising for junk food, and yet a substantial fraction of us manage to ignore that and eat healthy. I hope machine predictions that try to nudge us face a similar fate: predictions that look down on us will be indistinguishable from low-quality ones. The quality and tone of predictions will become an important part of the brand identity of the company emitting them. Do you really want to be the company known for peddling fatty, salty food, legal addictive substances, and predatory loans?

A tangential point: naming matters. Statistical inference, which is what machine learning really is, is a less scary name. “Statistical inference” conjures images of bean-counters with taped glasses; “machine learning” of an inevitable army of terminators.

Circling back to the question I posed above: is your life really describable in a few kilobytes in a predictive model? No. A model is a probabilistic abstraction. You, and your life, are not a sequence of probabilities, in much the same way that hiking a trail is a much richer, “realer” experience than tracing your way through a detailed trail map.

For example: You can chop off the the trailing “on” from “onion” and still form the word by wrapping around to the starting “on”. In other words, you can write the letters “o”, “n”, “i” in a circle, in that order, and form the word “onion”.

I say that a garland word is of degree n if you can do the above with the last n letters of the word.

It turns out that garland words are fairly rare. In the English dictionary that came installed on my Ubuntu machine, there were 99171 words, only 199 words were garland words of degree 2 or above. The highest garland degree was 4.

The exact distribution was: there were only 3 words of degree 4, 32 of degree 3, and 164 of degree 2.

The degree 4 garland words were: “hotshots”, “beriberi” and “abracadabra”.

This question comes into stark relief if you look at what a modern language or VM runtime does: among other things, it multiplexes resources, and protect access to them. But that’s also exactly what the OS is supposed to do!

The paper riffs on an earlier idea from Dan Ingalls: “An operating system is a collection of things that don’t fit into a language. There shouldn’t be one.” The central problem tackled in the paper is that of having more generic object bindings in the OS, as opposed to the “static” filesystem bindings for most operations. Kell uses the following example:

Suppose we wish to search some object for text matching a pattern. If the object is a directory tree of text files, grep -r does exactly that. But suppose instead our directory tree contains a mixture of zipped and non-zipped files, or is actually some non-directory collection of text-searchable objects, such as a mailbox. In Smalltalk parlance, how can we make these diverse objects “understand” (via a collection of interposed objects) the messages of our grep -r (and vice versa)?

This would indeed be elegant. But I’m afraid the engineer in me isn’t terribly excited by that.

“All problems in computer science can be solved by another level of indirection” is a famous quote in computer science. The next sentence is much lesser known: “But that usually will create another problem.” What Kell is proposing adds flexibility in binding by introducing indirection.

Indirection is controlled by some sort of configuration the binds each side of it. And configuration implies a lot of added complexity. The UNIX model of “everything is a stream of bytes and can be treated like a file” might be simple and unstructured, but that is precisely what enabled a multitude of structure to be built on top of it.

The only unit of time that matters is heartbeats. Even if the world were totally silent, even in a dark room covered in five layers of foam, you’d be able to count your own heartbeats.

When you get on a plane and travel you go 15 heartbeats per mile. That is, in the time it takes to travel a mile in flight your heart beats 15 times. On a train your heart might beat 250 times per mile…

… If we are going to ask people, in the form of our products, in the form of the things we make, to spend their heartbeats—if we are going to ask them to spend their heartbeats on us, on our ideas, how can we be sure, far more sure than we are now, that they spend those heartbeats wisely?

Turns out, heartbeats are a great unit of time for another reason: all animals have approximately the same number of heartbeats in a lifetime.

With each increase in animal size there is a slowing of the pace of life. A shrew’s heart beats 1,000 times a minute, a human’s 70 times, and an elephant heart beats only 28 times a minute. The lifespans are proportional; shrew life is intense but brief, elephant life long and contemplative. Each animal, independent of size, gets about a billion heartbeats per life.

I find that deeply poetic.

Most reactions to this have been along the lines of “look at these rich powerful elites thinking they can up and leave and do without all the rest of us!” I want to focus not on the politics behind this, but the motivation.

These ideas are getting attention now because rich and powerful titans of tech are espousing them. But these are the secret fantasies of every geek, rich or not, powerful or not.

Geeks have deep isolationist tendencies. They prefer losing themselves in sci-fi worlds to hanging out with friends. They build an early affinity with machines and programs. They sense that programming is the ultimate in world-building, and what’s more, it’s executable! They build elaborate towers of abstraction. They are drunk on the clean rationality of the world that the machine and they have built and understand.

Eventually, they secretly wish for a more ordered world. A world that is not messy or unpredictable. A world where one might have to work hard to establish the priors, but once they are, everything follows.

It is telling that one of the central activities in programming is abstraction. Abstraction is nothing but taking a jumble of intricate detail, putting it in a tightly sealed box, and then poking very small, well-defined holes in the box to interact with that jumble. So clean! So much better!

But every abstraction is leaky. Ints carry the baggage of their bit-width. In a perfect world, every int would be a BigInt with unlimited range. But we live in a world where it matters whether ints fit in a hardware register or not. Networks carry the baggage of not being perfect links. In a perfect world, a link between two machines would be an immutable, always-available channel for sending messages between them. But we live in a world where wires break.

This is the central tension in the mind of every geek. The constant need to abstract, to order, to rationalize, pitted against the necessary realization the world and entropy will eventually thwart you.

Secession is simply the same need for order and abstraction carried into the geek’s mental model for society.

I wrote previously:

The world of humans is messy, unpredictable, and—this is the part that infuriates you the most, the part you simply don’t get, the part that will forever make you an outsider—unreasonable. Humans are unreasonable. Your whole life, you’ve trusted one thing to save you. And saved you it has, again and again, without fail. That one thing is reason. Computers are a tool of reason, to be handled by people who obey the code of reason. People who are involved with computers, they’re reasonable, just like you. You all share a deep, unspoken bond. You’re all worshipers in the church of reason.

Please don’t hold that against us.

The problem is that all our modern computing infrastructure is written in a language, C, that was designed to be a thin layer over assembly and hence has no memory safety. A memory-safe language, which is to say, any modern post-C/C++ language, still allows writing buggy code that can be exploited–for example, look at the raft of SQL injection attacks in Java backends–but the bar for exploitation is much higher and the space of possible attacks much smaller when an attacker can’t turn array dereferences into a way to read and write arbitrary memory addresses.

Type and memory safe languages have existed for a while now. Language based security and static analysis is still a thriving research area, but it is also mature enough that a broad set of results and techniques have been found to be generally useful, many of which have even made it into mainstream languages and tools. Why then are we still in this state?

The two immediate answers that come to mind–improving the safety of C/C++, and writing infrastructure in safe languages–are, upon closer inspection,riddled with complexities.

Making C/C++ memory safe is possible, but it is not as simple as flipping a switch. There is a price to be paid in performance, of course, but even if one were willing to pay that, there might be scores of subtle behavior changes in code made safe this way. There is probably tons of code out there that depends on unsafe behavior.

Writing SSL in a safe® language is possible. But everything that would want to link it (browsers and web servers) is written in C. One could use cross-language calling facilities, but that is rarely clean due to impedance mismatches, and of course, that still leaves the calling code memory-unsafe. You’d have to rewrite all the net’s critical infrastructure in a safe language and runtime. That’s wishful thinking.

When physical infrastructure decays it is visible. A new road uses the latest knowledge and practices for building roads, not those from thirty years ago. Neither is the case for our computing and network infrastructure.

In a strange inversion of the commonly held belief that hardware is hard to change and software is easily changed, some burden for safety might eventually fall on the processor. One example was when CPUs started supporting the no-execute bit to prevent code injection attacks. I wouldn’t be surprised to see more hardware features along those lines.

Relevant links:

• Great overall explanation of the bug
• John Regehr on using static analysis to find Heartbleed
• Coverity on how they tweaked their analysis to find Heartbleed

I particularly looked forward to the chapter titled “Racing With The Machines: Recommendations for Individuals”, in which they suggest what kind of work will be left for humans to do, and what skills we should try and develop to stay ahead of automation. I really wanted to know if they had cracked the nut of how we would stay gainfully employed. I was disappointed.

The the authors looked at prior attempts by Levy and Murnane that tried to predict limits on what activities would still require humans. They said that computers are good at following rules and bad at pattern-recognition, giving the example of driving a car as being too cognitively and sensorially complex to be automated. As the authors say in the book: “so much for that distinction.” Then they go right ahead and make their own prediction of the next frontier that computers will be unable to cross:

Our recommendations about how people can remain valuable knowledge workers in the new machine age are straightforward: work to improve the skills of ideation, large-frame pattern recognition, and complex communication instead of just the three Rs. And whenever possible, take advantage of self-organizing learning environments, which have a track record of developing these skills in people.

The biggest problem with this advice is that it is skating to where the puck is, not where it is going to be.

The example they lean on the most is freestyle chess where teams of humans and chess programs compete against each other (hence the title of the chapter). It turns out that the combination of a chess program guided by a human is more powerful than either alone. Freestyle chess is also a central example in Tyler Cowen’s latest book. Unlike Brynjolfsson and McAfee, Cowen wonders if this is just a transitional phase, and if humans will ultimately not add any value in this pairing.

Their recommendation about “ideation” and “large-frame pattern recognition” is not concrete enough. What does that mean specifically for someone choosing a college major today? And more importantly, can we be sure that those activities will remain out of the reach of computers by the time they graduate?

The debate about whether human thought is computable is an open one, but the vast majority of human cognitive activity does not happen anywhere near that threshold. In an ironic similarity to the diagonal cut, perhaps the only people secure in their isolation from automation are mathematicians probing the bounds of what is computable, and how.

But each one of the rest of us has to wonder if, within our lifetimes, our jobs are automatable. I program for a living, and while a good fraction of what I do is intellectually challenging, there is also some that makes me feel like just an operator.

Many non-technical authors think of Kasparov losing to Big Blue as a major milestone in AI, but that was largely due arriving at a point in time that Moore’s Law delivered enough computing beef to “solve” an exponentially complex game like chess. A more meaningful threshold would be when insight can be computed. For example, could a computer propose an algorithm with the simple elegance of Quicksort?

“Running with the machines” is a temporary strategy at best. That is simply the halfway house between humans doing something and it being fully automated. A more accurate phrase would be “run by the machines”, because cheap human labor is crowdsourced for the kinds of problems that are just barely outside a computer’s (current) reach.

I see two strategies for staying ahead of automation:

The first is to be the one doing the automation. In other words, to be a programmer. (Massive disclaimer: of course I would say that, being a programmer.) More bluntly, be the hunter instead of the hunted. The problem is that not everybody is able or willing to do that.

The second strategy is to be a doctor of machines. Large computing systems show organic behavior, and tending to them often requires the same mindset and behaviors as a doctor diagnosing and tending to patients. I like to draw an analogy to cities. A (relatively) small number of people initially construct the infrastructure (pipes, wires, roads), but a much larger number of people maintain them continuously.

We will have interesting lives.

Some thoughts:

1 - Net connectivity does not necessarily mean “thin client”. I can use net connectivity for data and still do a lot of processing on the client.

2 - The server/client power ratio doesn’t matter unless I’m actually trying to solve those large problems.

3 - There are some problems that need large machines and there are also some problems that could fit on either side. We’re not trying to figure out whether web search is going to be running on the phone. The real question is where the small things run.

4 - I do use my phone and tablet when I don’t have net connectivity. They are not bricks (maybe this is different in Androidland). Most of the things I do aren’t big problems. Even some previously big problems that only fit on a big computer now fit on my phone (like maps).

5 - Bandwidth and storage keeps getting better. Latency and availability aren’t improving much. Bandwidth and storage makes the “sync all my data among all my devices” model work for a larger and larger subset of my data. My data is not growing as quickly as storage, so I can see all my data fitting on my phone within a decade.

For any given problem, do I ever want to give up low latency and high availability to have it run on the server instead of my client? Generally, not unless it’s a much better experience to compensate for the slowness and unavailability. For most everything I do (text editing, word processing, photo processing, web browsing, games, email, social networking), it’s hard to justify.

Unfortunately most of today’s client apps are based on the old model assuming no networking. You email files around if you want to share. I think we can do much better by using synchronization among multiple clients as a fundamental building block. Data and compute stay local but when you have connectivity you synchronize with other places the data is stored. Google Wave was a stab at this. OneNote too, I think. Git does this for programming. DropBox is built on this model but it handles only simple synchronization at the moment. They have the potential to build infrastructure for apps to synchronize in a more structured way (like Wave), but I don’t know if they’re planning to do this.

[one of the authors] has had experiences facilitating web-based video dialogue, and even though she sees it could have enormous potential when it works well, very often it does not. When we learn online, we are not together in one room, and we need to recognize not only the limitations of that, but the openness of its possibilities.

The strengths of online learning, especially in massive courses such as MOOCs, and especially for adult learners, might lie in their asynchronous interactive components.

Read the whole thing.

So why can’t they be completely automated? I mean that literally. Could we have software that carries out all those functions?

Software could propose new products within a design space. It could monitor the performance of those products in the marketplace, and then learn from the feedback and adjust accordingly, all while maximizing its value function. If the product is a webapp, most of these steps are already within the reach of full automation. For physical products, what’s missing is APIs that allow a program to order physical products, and move them around in the physical world.

A limited version of what I’m describing already exists. High-frequency trading firms are already pure software, mostly beyond human control or comprehension. The flash crash of 2010 demonstrated this. Companies that are centered around logistics, like FedEx or Walmart, can be already thought of as complex software entities where human worker bees carry out the machine’s instructions.

This happens naturally, because over time more and more of the business logic of a company becomes encoded in software. Humans still have some control (or so they think) but mostly what they’re doing is supplying parameters to the computation. A modern corporation is so complex that it does not fit in the brain of a single person (or a small number of persons). Software carries the slack.

The intermediate step to a fully automated corporation is one where tasks requiring humans are performed not by employees but are broken into micro-tasks and fulfilled by crowdsourcing (using, for example, services like Mechanical Turk).

Corporations do not scale, and eventually die. That’s because they scale sub-linearly. Their productivity metrics scale by an exponent of ⅘ on the number of employees.

I hypothesize that the management overhead which makes corporations grow sub-linearly is due to the limited information processing capability of individual humans. People at the top do not have local on-the-ground information: how are individual products performing, what are customers’ complaints etc. And the rank-and-file folks on the ground do not have the relevant high-level information: how does what I’m doing translate to the value that the corporation as a whole seeks to maximize? In fact, the the flow of value and information is so complex that employees have pretty much given up on determining that relationship, and know of it only at a macro P&L-center level.

An algorithm will have no such problems with acting on both global as well as fine-grained local information. In fact, I suspect that the more information it gets to act on, the better decisions it will make, making automatic corporations grow super-linearly.

One objection to being a purely automatic corporation might be that it precludes visionary, groundbreaking products, because it resembles incremental hill climbing and is susceptible to get stuck in local maxima. But a visionary, groundbreaking product is (initially) indistinguishable from picking a new random point in the product space, and hill-climbing from there. There is no reason why the automatic corporation couldn’t allocate some fraction of its resources to exploring new points in the space completely unrelated to its current neighborhood.

The CEO of an automatic corporation will be a devops engineer: fixing software bugs, writing “features” (i.e. new ways for the corporation to behave), watching performance dashboards (imagine all the pretty graphs!), and providing some vestige of human input to tune parameters used by the software. Eventually they can take their hands off the steering wheel, having configured everything to run on auto-pilot, and set up alerts to page their phone if something really goes wrong.

Automatic corporations will start within narrow domains, where the product design area and value functions are relatively simple. But they will gradually move upmarket. This is a familiar pattern in the dissemination of most new technologies. There will be a smattering of small automatic corporations by the end of the decade.

What I really need to keep track of are not emails and docs and appointments, but projects.

Imagine these emails/docs as pebbles, and each project as a collection of pebbles. This project is red pebbles. That one is blue pebbles. Yet another one is yellow pebbles. Then imagine throwing all of them on the floor. Then imagine being asked “can you construct for me a coherent state of all the yellow pebbles?” That’s how it feels trying get a high-level view of a project from all these pebbles flying around.

I feel like email and shared docs are like assembly language, and we need an abstraction above them. The analogy is fitting, because a high-level language is restrictive: you can do more things and have more control in assembly. But with the restriction comes freedom to express more complexity more succintly.

The problem with every project management solution that I’ve seen is this: it assumes it is the only thing in the world. “If you use this tool, and only this tool, and capture every little detail as it happens, then this tool will give you a coherent state of the world, yaaay!” Of course, life never works that way.

I’m not saying this is an easy problem to solve. I don’t even know what a first approximation of the solution would look like. But I do know there an opportunity here.

But in a departure from his usually cold hard intellectual style, his recent piece about how the quality of light affects our work and our rhythms really tugged at my heart. A lot of that was because of the most powerful rhetoric device–nostalgia. Because I grew up in India in the 80s and 90s and also went through those load-shedding hours. It was inconvenience, boredom, excitement and respite all rolled into one.

Dark evenings without power were a special time for children. The candle-lit hours on porches and balconies were a strange mix of an ethereal kind of intimacy, beckoning darkness, and thoughts that retreated from both sunlight and electric lights.

You could do nothing useful during those hours. There was no TV or radio. Reading was difficult. Candle-lit meals tended to be either quick, simple affairs whipped up in semi-darkness, or leftovers. Families who turned the blacked-out evenings into family time generally sat out on the porch. Adults would use the time to tell family stories to children. Teenagers and some couples would stroll up and down the street, occasionally stopping to chat with neighbors. Younger kids would run around squealing and playing, seemingly possessed by the strange euphoria-inducing forces leaking in from another world. Or they would huddle together and try to scare each other with ghost stories.

Now, the target deployment platform is an app for a phone or tablet. But you still need a big desktop machine to develop them.

The first effect of this development-deployment divergence is that it cements the new position of the desktop as a specialized factory machine.

But another effect is that it raises the barrier to casual hacking. In the PC age a kid who wanted to write software just needed to download a bunch of simple tools and was off running. But today that kid would need to setup a complicated toolchain on a beefy desktop. And that too in an age when they think of desktops as something quaint, and might very well not even have one, because their “primary device” is just their phone or tablet.

Even the term “primary device” probably sounds funny to them. Growing up if someone asked me what my primary device was I’d laugh at them because my PC was my only device and it was all I needed or wanted.

So here’s my wish—and challenge. Make it so that a young person who gets the urge can code and build something on their phone or tablet. I suspect it will be easier with the extra screen real estate on a tablet, but the big problem is the same. Make it so that some of that infinite interstitial time that their phone soaks up can be filled with programming. What if at the end of that Apple ad the teenager presented his family with a cool app rather than a movie?

• first class functions
• lists and maps are part of the language
• succinctness

These features first originated in dynamically typed scripting languages like Python. (Well, they originated in Lisp, but then everything originated in Lisp.) Then they gradually found their way into statically typed compiled languages like C++¹ and Java. Some of the most anticipated features of Java 8 are lambda expressions and type inference.

There is a subtle distinction between succinctness and expressivity. Expressivity almost always leads to succinct code. My favorite example is list comprehensions. But succinctness is also about omitting unnecessary verbiage. A good example is type inference enabling the omission of spelling out types.

Go had all three from the get-go.

Two things that are not yet part of this common core are objects and concurrency. I almost added “object orientation” to the list above, but then Go’s model of defining structs and interfaces made me think otherwise. It reminds me of Niklaus Wirth’s philosophy of understanding programs by looking at their data structures first. As for concurrency, there are wildly different ways of expressing it, but I have a hunch that the CSP model adopted by Go (and Erlang) will prevail. Maybe in another few years there will be five items on the list of common language features.

This is a good thing, because it looks like we’re gradually finding our way towards abstractions and constructs that are proving to be widely useful and broadly applicable. It is also rendering irrelevant many of the PL community’s fiercest debates. The agitation over static vs dynamic types starts fading when a C++ or Java code snippet with range-based for loops, lambdas and omitted type declarations looks very much like a Python or Ruby one. I was also surprised to see that while Go is very much a systems language, in chatting with other programmers I found Go eating more share from Java and Python than C++.

I call myself a protopian, not a utopian. I believe in progress in an incremental way where every year it’s better than the year before but not by very much—just a micro amount. I don’t believe in utopia where there’s any kind of a world without problems brought on by technology. Every new technology creates almost as many problems that it solves. For most people that statement would suggest that technology is kind of a wash. It’s kind of neutral, because if you’re creating as many problems as it solves, then it’s a 50/50 wash, but the difference in my protopian view versus, say, a neutral view is that all these new technologies bring new possibilities that did not exist before, including the new possibility of doing harm versus good.

How did I navigate? In the days (weeks?) before the trip, I drew maps on paper of our entire route, with all the places we wanted to see. I used Google Maps as the source. It was probably tens of pages of hand-drawn maps. It was tedious, but it did have advantages:

• I omitted unnecessary details. Just the streets I was going to be on.
• I highlighted relevant details: landmarks, distances.
• The canvas was large: letter paper.
• Preparing the maps and planning the trip was fun.

I wish I had saved that paper, because it would be a memento as evocative as the pictures from that trip.

(In 2001, LineDrive tried to address some of these issues by automatically generating driving directions in a style similar to hand-drawn maps.)

I tell you this story not to reminisce about my travels (well, maybe a little bit) but to point out that analog artifacts still have affordances that give them advantages over digital devices. Even today, for complex multi-hop trips I prefer to hand-draw maps on paper.

I want my digital interfaces to be more organic and analog, and my analog experiences to make it into my digital corpus effortlessly.

Star Trek is an expression of the idea that technology cleanses people. It is technology manifested outwards to people and society. And technology is clean and rational.

Star Trek depicts a clean world. Not just physically clean, but also emotionally and socially clean. There are no politics. (Did Picard ever have to contend for limited resources with other Star Fleet captains?) There is no heartbreak or jealousy. There is no class. There are no vices, no addictions, no crime. The Enterprise does not have a dark, seedy corner. There is no soul-searching. It’s telling that the non-human Data is often the most reflective character. Good and evil are clear. Star Fleet and everyone it in is good. All problems and all evil comes from the outside (Q, the Borg, other civilizations etc etc).

Babylon 5 is an expression of the idea that human nature is independent of technology. It is technology as one part of people and society. And people are messy and irrational.

Babylon 5 depicts a world of humans with all their failings. In it, technology has brought us far into space, but we are all still humans. Life is still messy and unpredictable and hard. Politics, within the humans, and among the space-faring races, is central. Society is stratified. There are haves and have-nots. It is not always clear which side is “right”. People experience joy, sorrow, heartbreak and the full range of human emotions. They battle demons and addictions.

The problem with the old thin client model was the assumption that processing power was scarce. In fact, Moore’s Law and the rise of ARM has made the exact opposite the case – processing is abundant… …Thus, over the last few years as the number of fat clients has multiplied – phones, tablets, along with traditional computers – the idea of a thin client with processing on the server seems positively quaint; however, in the context of our data, that is the exact model more and more of us are using: centralized data easily accessible to multiple “fat” devices distinguished by their user experience.

Calling today’s phones and tablets “fat” is, I think, quaint. It is an easy trap to fall into. They are many orders of magnitude more powerful than those a decade ago. If the Palm Pilot was a thin client, certainly a modern iPhone cannot also be called “thin”.

But the thin client computing model was never about the absolute processing power, or lack thereof, of the clients. The crucial factor was the relative difference in processing power and storage between the client and the server. No matter how powerful the device we’re holding in our hands is, the warehouse-scale computer on which it depends is many, many more times more powerful. And without knowing what datacenters two decades ago looked like, I think it’s safe to say that the server/client power ratio has only increased over time, and is likely to continue increasing.

A smartphone or tablet’s impressive computing power goes largely into pushing pixels around, and giving us a great experience by responding quickly. It would be useless without all the backend services (app stores, cloud storage, social networks… the list is endless) it derives its usefulness from. A phone or tablet without net connectivity if pretty much a brick. And so, by definition, it “depends heavily on some other computer (its server) to fulfill its computational roles”.

What was I doing? What do I need to do next? These are the most important questions when I’m working. And modern interfaces make it easy to lose track of them when I’m in the thick of it, and downright hard to answer them when I try to come back after a break, or an interruption.

The contours of my attention are varied and varying. They have peaks and valleys and plateaus. They are changing, rising and falling. Some things are central, some peripheral. Some things are important, some urgent, and some whimsical. Some peripheral things might be important. Some whimsical things might become central.

The modern UI does not know about the contours of my attention. It is a vast plain, with no distinguishing marks.

(Note that this is a different problem than that of being distracted, and being on a device where distraction is so easy. Not that I don’t suffer from that problem too but that’s a whole different story.)

I’ve taken to going analog. Writing things down. I got a couple of nice fountain pens that lightly tickle my nostalgia bone and that make me enjoy the sensual pleasure of putting wet ink on smooth paper. Underlines. Doodles. Boxes. Large writing. Small writing. They all tell a story. They all anchor thought, and recollect it.

Rather than the keyboard driving, I let the paper drive. I write tasks and designs and box and arrow diagrams. Then I go do one little thing on the computer. Then I come back to the paper. And so on. This is very similar to the idea of having an analog desk and a digital desk.

This is the next challenge in designing UIs: making them recognize and adjust to the contours of my attention.

• the documentation is fantastic. Having a tutorial where one can execute (and edit!) the examples on the webpage itself really helps.
• the syntax favors terseness. It is a relief after the chattering verboseness of something like Java. Types don’t need to be always declared and are inferred wherever possible. There are a few quirky choices: I didn’t think I’d see “:=” after I moved on from Pascal. And the biggest breakthrough of all: the automatic formatter “gofmt” saves Go programmers quintillions of hours in avoided debates about formatting style and micro-futzing with spacing in the editor.
• concurrency is part of the language in the form of channels and goroutines. About time too, since we’re in the 21st century and threads cannot be implemented as a library(or at least, it’s always clunky when they are).
• like Python, the cognitive size of the language is small enough to fit in one brain without constantly thrashing out. I write Go in an editor (Emacs), not an IDE, and that works just fine. With Java there was a heavy productivity hit to opting out of something like Eclipse and auto-complete. I felt like the IDE was an earth mover for code.

If one is writing a new system (especially a backend) from scratch that doesn’t have to call existing C++ or Java code (or only calls them via RPC), then Go is a pretty compelling option right now. That is great after a decade of no other viable options.

I never thought I’d say this, but its exhausting. Even worse, it’s boring.

Seeing as I make my living in the field, this conundrum spurred some soul-searching in me. I don’t want to be the software engineer who bitches and moans about tech.

My first computer had a monochrome (green!) monitor with a resolution of 720x348 pixels. Moving up to 4-bit color was a huge step up. Fast forward to today, with millions of colors and resolution indistinguishable from natural surfaces, what remains for flat displays to accomplish? The next interesting thing to do is take displays to places they haven’t been to yet, like inside your eyes.

Once displays crossed the threshold of human resolution, they were essentially “done”.

Something similar has now happened with computers. For the vast majority of users, specs like amount of RAM and CPU speed simply don’t matter any more. At some level, even individual computers don’t matter any more. What’s more interesting is what you can build with giant warehouses full of them. Capabilities like Google Now and Siri (powered by warehouse computers) have pushed what we can do with phones more than a faster SnapDragon processor.

And that’s the source of my ennui with CES. A lot of what they’re showing is pushing past the human resolution threshold. More of more. We need to go to new places, even if it is with less.

If anything, I think he oversold parts of it, but that’s probably because someone who hangs out in my corner of the internet already knows that yes, we obsessively and compulsively use our connected devices, and yes, it hurts our focus and even our relationships, and yes, there are tools like Freedom and WriteRoom that claim to help.

I also wish he had delved a bit more into a word from the title of the book: addiction. About how we use our devices and connections in addictive ways, with no tangible benefit. About how, just like addiction to other substances, one needs to come face-to-face with the empty void that’s sucking them in before they come to their senses. How is a compulsive itch to swipe-to-unlock your phone different from a yearning for a shot of cocaine? You would’d sneak out a flask of whiskey from your jacket and take a swig in the middle of a meeting, but it’s completey acceptable to peek at your cellphone.

Reading the book reminded me and made me go back and read this manifesto for calm computing, written back in 1996. (At the risk of sounding pedantic, I’ll say that was a glaring omission from Pang’s citations). In it, the authors foresee our current crisis of attention, more than a full decade before the onslaught of net-connected smartphones. And they correctly identify the need for technologies that, as they say, “both encalm and inform.” The central principle they identify is that our interfaces need to show us both a center and its periphery, and appropriately move items between them. That’s what makes it not overwhelming when we perceive the natural world, or when we walk down a busy city street. Things naturally flow from the periphery to the center of our attention and back. Indeed, I know of no modern prevalent UI that does this, nearly two decades after the authors wrote this. Modern UIs are what I like to call attention-flat: everything is arranged on the screen either in a way that pulls attention equally (many windows visible) or in a way that completely obscures the periphery (one window maximized, hiding all others).

That makes me think there’s much more we can do on the technology front, particulary with UIs, to tackle the issue.

Great read. Recommended.

• Computer Science PhD treands
• The laziness of synchronous work
• What is life like for PhDs in Computer Science who go into industry?
• Empirical analysis of programming language adoption
• Future programming

This was also the year I started a newsletter where I send out occasional tidbits of raw thought not ready for posting yet.

Every blogger has posts where they bleed, but that don’t get enough hits. Mine were:

• Some of my peeves with the idea of “productivity.”
• My insecurities about my job being automated away
• The Hydra problem of balancing “life” and “work”
• Startups aren’t all they’re cranked up to be: says the comfortable corporate citizen
• On being torn between the digital and the analog

There is endless debate about the characteristics of each generation, but how can one assign characteristics to something as broad as an entire set of humans born in three designated decades? But one can at least try to assign characteristics to their times.

I feel like part of neither Gen X nor the millennials. Gen X feels too old, more like my parents’ generation. And most millennials feel too young.

I feel like part of the analog-digital generation. I’m calling it Generation AD. The generation before was firmly analog. The generation after are digital natives. Generation AD is the confused one in the middle.

I define Generation AD to consist of those who spent their early youth (approximately up to high school) living a largely analog lifestyle, and transitioned to a largely digital one thereafter.

The internet had not caught on when they were young. There wasn’t a cellphone in everyone’s hands. No screens in the classroom! They had to call their friends on landline phones, make concrete plans about when and where to meet, and then go there. They learned cursive handwriting in school. They were taught that penmanship was important.

And then, by the time they were in their late teens and early twenties, a giant switch got flipped in the span of a few years. The internet exploded. Mobile devices proliferated. Digital took over. Typing replace handwriting. Cellphones meant everything could be planned in real time and nothing needed to be planned of ahead of time.

Generation AD deeply understands the advantages of being cleanly digital, and yet feels a nostalgic pull to analog things like pens and paper. This results in Frankenstein workflows like making heavy use of notebooks… and then scanning them for posterity. Bits, after all, are much lighter than atoms, and moving a hard disk is much easier than lugging boxes of notebooks.

Generation AD is the last and only one familiar enough with both sides of the digital/analog divide to truly reason about each. Gen X is analog at heart and puts up with digital life mostly because it has to. Millennials skipped analog altogether. Neither are in a position to produce nuanced critiques of our digital lifestyles. Only Gen AD knows which nooks and crannies analog fills better, and hence truly appreciates those where going digital has more impact. Gen AD consists of people who remember concepts or quotes from books spatially (“it was on the top of the left page, somewhere near the middle of the book”), but who will gladly ditch the dead-tree version and sling their tablets when they travel.

We will be perpetually stuck in the middle.

in an age of personal technological revolution, we all need a more explicit philosophy for adopting tools. Without this clarity, we run the risk of drowning in a sea of distracting apps and shiny web sites. My philosophy — to only adopt tools that solve a major pre-existing problem — has served me well. I use e-mail, for example, because the ability to communicate asynchronously with people around the world is quite important for my work. E-mail solves this problem. I don’t use Twitter, however, because the ability to have short, casual interactions with many people I don’t know well is not that important to my work.

He addressed some more arguments in a follow-on post.

On the face of it, that’s a reasonable position to take, especially for an academic whose job is to think deep thoughts and solve big problems. But it also strikes me as showing a lack of playfulness, which is important to cultivate, even if only in small doses. It might also miss out on entirely new ways to seed and develop academic ideas.

Take, for example, this account of how an idea started with a tweet and ended up becoming a full journal paper:

My experience with the germ of an idea shared as a Tweet at an academic conference that became a blog post, then a series of blog posts, and (eventually) a peer-reviewed article is just one example of the changing nature of scholarship. From where I sit, being a scholar now involves creating knowledge in ways that are more open, more fluid, and more easily read by wider audience.

Newport is making a stand for the lone academic solving problems all by himself, and one who is hence fiercely protective of his solitary time. But the era of the groundbreaking single-author paper (or any type of single-author paper) is long over, and science currently is advanced by vast teams.

Kevin Kelly talks about how even the Amish, famous for being luddites, have a process for evaluating and adopting new technologies:

The Amish have the undeserved reputation of being luddites, of people who refuse to employ new technology. It’s well known the strictest of them don’t use electricity, or automobiles, but rather farm with manual tools and ride in a horse and buggy. In any debate about the merits of embracing new technology, the Amish stand out as offering an honorable alternative of refusal. Yet Amish lives are anything but anti-technological. In fact on my several visits with them, I have found them to be ingenious hackers and tinkers, the ultimate makers and do-it-yourselfers and surprisingly pro technology.

Ten years ago when I was editing Wired I sent Howard Rheingold to investigate the Amish take on cell phones. His report published in January 1999 makes it clear that the Amish had not decided on cell phones yet. Ten years later they are still deciding, still trying it out. This is how the Amish determine whether technology works for them. Rather than employ the precautionary principle, which says, unless you can prove there is no harm, don’t use new technology, the Amish rely on the enthusiasm of Amish early adopters to try stuff out until they prove harm.

We all need a small budget of time for playing and evaluating tools with an open mind. We might discover uses we hadn’t thought of, and solve problems we didn’t know we had.

Software writers themselves don’t seem immune from the new de-skilling wave. His comments point to the essential tension that has always characterized technological de-skilling: the very real benefits of labor-saving technology come at the cost of a loss of human talent. The hard challenge is knowing where to draw the line—or just realizing that there is a line to be drawn.

The comment thread on his post has some great discussion. You should go read it. In it, Carr raises two more questions, which I will try to address here.

Climbing to higher levels of abstraction

re: “each advance opens up more new opportunities than it removes” This is a common defense of automation in pretty much all fields. (See the quote from Alfred North Whitehead in my Atlantic piece.) And it’s a good defense, as it’s very often true. But there are also a couple of counterarguments:

At some point, as the capabilities of automation software advance, the software aid begins to take over essential tasks – sensing, analysis, diagnosis, judgment making – and the human shifts to more routine functions such as input and monitoring. In other words, with the automation of skilled work there’s a point at which there are no “higher-level tasks” for the human to climb to.

In the world of software there is always a higher level to climb to. But an individual software engineer has to have both foresight and will to do it.

For example, not so long ago, we were building regular CRUD web applications by hand every time. Soon common patterns emerged, and were codified in web frameworks like Ruby On Rails and Django. The level of abstraction for coding web apps was lifted. The interval of interest is between web apps becoming common but before a large part of the boilerplate was extracted into frameworks. If you were a web app developer during that period, the repetitive nature of coding web apps should have made you feel that a large part of your current job was ripe for automation.

As in every arena where automation takes over, in software engineering too the pattern is familiar: as an activity becomes widespread, the repetitive and “non-thinking” parts of it are recognized and extracted. In the case of industry, they are extracted into a machine or robot. In the case of software, they are extracted into a layer of abstraction or framework. There is an uncanny valley just before that happens when workers get the feeling that they’re just going through the motions.

The alphas are those who extract common repetitiveness into automation. The betas are those who raise their skills to adding value on top of the newly automated tasks. Everyone else gets left behind.

Cutting across layers of abstraction

Lower-level tasks may be seen as mere drudge work by the experienced expert, but they can actually be essential to the development of rich expertise by a person learning the trade. Automation can, in other words, benefit the master, but harm the apprentice (as R. Carey suggests in the preceding comment). And in some cases even the master begins to experience skill loss by not practicing the “lower level” tasks. (This has been seen among veteran pilots depending on autopilot systems, for example.)

I think this is a real danger. Teaching new students high-level languages is great to get them started, but for for true systems-building one needs to understand the whole stack.

The thing about computer science is that all abstractions are leaky. We can’t use our abstractions without understanding their underlying implementation and limits. This happens at every layer. An integer shows its implementation as a 32-bit word on the microprocessor when adding to MAXINT wraps around. A garbage collector in a VM frees us from manual memory management, but shows up when the app experiences unpredictable pauses. Even the CPU, the bedrock abstraction, leaks its implementation details when things like speculative execution and branch prediction affect performance. The examples are endless.

The other thing is: as Bryan Cantrill points out, failing and pathological systems are the ones which truly teach us. They lay bare our tower of abstractions. One has to understand all the layers and their implementations to debug them.

So becoming comfortable and knowledgeable in just one layer is brittle, and that brittleness is exposed the moment one has to solve or debug a significant problem.

For the apprentice, this means learning and working with as many different layers of abstraction as possible. For the more experienced, it means not letting the day-to-day job in one layer make you rusty with the others. These are both challenges.

Peeking into the future of automating software engineering

The peculiar thing about computing and software is that it can recursively coil around to build upon itself. Large amounts of code are used as data to gain insight into the process of writing and changing software. Sophisticated algorithms can learn from this code/data to automatically generate changes and fix bugs in existing software. And the fledgling field of computational creativity can ingest the mountains of now digitized human creative output to guide the “creativity” of an algorithm. A paper from the 80s was prescient when it claimed that software processes are software too.

So, more computing power and more sophisticated algorithms and more data allow you to build even more powerful computers and even more sophisticated algorithms. This is unlike physical industry. A tall building doesn’t help in building even taller buildings.

It is in this line of thinking–connecting the individual programmer to a datacenter’s worth of data, analysis and computational power–that I think the future of automating the process of writing software lies. My hope is that the combination of human and machine will be more potent than either alone, much like advanced chess.

His latest piece in The Atlantic talks about the perils of automation, and what it means for human capabilities and skill:

Psychologists have found that when we work with computers, we often fall victim to two cognitive ailments—complacency and bias—that can undercut our performance and lead to mistakes. Automation complacency occurs when a computer lulls us into a false sense of security. Confident that the machine will work flawlessly and handle any problem that crops up, we allow our attention to drift. We become disengaged from our work, and our awareness of what’s going on around us fades. Automation bias occurs when we place too much faith in the accuracy of the information coming through our monitors. Our trust in the software becomes so strong that we ignore or discount other information sources, including our own eyes and ears. When a computer provides incorrect or insufficient data, we remain oblivious to the error.

How automation is creeping higher up the labor stack to so-called white-collar knowledge jobs, especially programming, has been on my mind for a while. Modern IDEs are getting “helpful” enough that at times I feel like an IDE operator rather than a programmer. They have support for advanced refactoring. Linters can now tell you about design issues and code smells. The behavior all these tools encourage is not “think deeply about your code and write it carefully”, but “just write a crappy first draft of your code, and then the tools will tell you not just what’s wrong with it, but also how to make it better.”

I’ve been on both sides. There are times when I fire up Eclipse just so that I can auto-complete my way to put out some code now. There are other times when I stare at the stark screen of a text editor (not IDE), the blinking cursor accosting me: “what the hell are you going to do now without all your fancy crutches?” But after the initial stab of fear and helplessness, I revel in the freedom of it, and savor the delicious rebellion of writing code without an IDE nagging me with suggestions and hints and corrections at every damn keypress like a 21st century Clippy from a horror movie.

Am I arguing for primitive tools? No. What I’ve described in the previous paragraph are my personal feelings while writing code. The tools (at least, the good ones) encode the knowledge and hard-won lessons of an entire army of programmers. They often point me to issues I’m blind to while writing code because I’m in such a rush to just make the damn thing work. In aggregate, they lead to a cleaner codebase.

So what am I saying then? I’m saying that we should let the tools help us without becoming crutches, to let us write sharp code without dulling our minds. That sounds paradoxical. It is a subtle mental stance one takes towards one’s work, tools, and output.

The most guru programmers I’ve seen have this paradoxical zen-like stance: they use the tools to the full extent possible (because, well, that’s what they’re there for), but understand enough about what those tools do and how they work to be perfectly productive even without them. It reminds me of the concept of moh, and non-attachment, while being an active participant.

And when it’s finished, I feel somewhat tricked and uncomfortable. I can’t find one single glaring problem, but the final feeling I’m left with is that there just have to be a ton of simplifications and glossed-over complexities.

This is a problem with pop sci (or pop sociology) in general, but particularly with the specific strain as practiced by writers who themselves do not have a technical background.

In a recent Longform podcast (transcript) Gladwell says that his central concern is the richness of the story he is telling. He gushes with admiration for Michael Lewis because he can tell a story without citing papers.

My great hero as a writer is Michael Lewis. I just think Michael Lewis, believe it or not, is the most underrated writer of my generation. I think he is the one who will be read 50 years from now. And I think what he does is so extraordinary, from a kind of degree of difficulty standpoint. The Big Short is a gripping book, fascinating, utterly gripping book about derivatives. It blows me away how insanely hard that book was to do, and it’s brilliant. The Blind Side, I think, it might be the most perfect book I’ve read in 25 years. I don’t think there’s a single word in that that I would change. I just think it has everything. But he uses no science, right? Very little. It’s all story. But he does more work in his stories, makes much more profound points than I do by dragging in all these sociologists and psychologists. He’s proved to me that, if you can tell a story properly, you don’t need this kind of scaffolding. You can just tell the story. And so, I’ve been trying, not entirely successfully, but trying to move in that direction over the last couple books.

That’s the classic narrative fallacy–seeing a connecting narrative where none exists.

As I’ve said before, researchers and scientists themselves are too shy to write in this manner, but

Authors want to believe. They want to find the parts of a dense technical paper that help them build a colorful story and support a somewhat related generalization.

I’d like to think that I can understand something by reading summaries of it or narratives about it, but time and I again I keep realizing that there is no substitute for the original source material.

The opening paragraph of the book puts it best:

Instead of solutions for problems, programmes for solutions… for no problem is there an absolute solution. Reason: the possibilities cannot be delimited absolutely. There is always a group of solutions, one of which is best under certain conditions.

For example, this logo for a music shop adapts to various paper sizes and layouts, while conveying an identity.

He also talks about how to think of design as choosing points in space of axes, where the axes are things like color, layout, type, size, contrast etc. That was the inspiration for my earlier post on automating web design.

Russ Roberts, the host, asked Bhagwati, “…What changed in 1991?”, to which he replied:

…the Prime Minister himself … said, look, I have lots of my own family abroad, sons and daughters and nephews and nieces, and this is true virtually of everybody in the upper classes in India–they have family abroad. They keep coming back and saying: How come we have such idiotic policies? With such enormous restrictions on diversification, on capacity expansion, etc., etc., which are driving us into the ground? And so, the diaspora effect is what I call it. A lot of young people coming back said: You really cannot have this. Because India is really losing rapidly its position in the world economy. Because if you are not performing well, nobody is going to pay attention to you. And the second thing I think was that increasing as people went … abroad and they would find that nobody took India seriously. So the Indian politicians and bureaucrats were increasingly running into situations where they were simply disregarded and looked down upon.

That might be a factor, but it strikes me as a weak one. The much larger factor was that the Gulf War brought cable TV to India, and with it, the culture and mores of the West. When you go from decades of one state-sponsored TV channel that broadcast eight hours a day to dozens that run 24 hours a day, the impact is enough to move a nation. If you grew up in India during the 90s, there is absolutely no question about the impact of cable TV. Like I wrote before:

And when we were not watching the military-industrial complex peacocking, we were soaking in its cultural counterpart. Baywatch. MTV. Moonlighting. We didn’t even have to filter the crap. Only the hits were re-broadcast. Suddenly our culture seemed parochial. Suddenly our things seemed archaic, clunky.

To that end, I cooked up some simple Javascript to randomly pick style choices from a given range and apply them to a page. The style choices for now are simple: text width and alignment, typefaces for headers and body text, type scale and color pallette.

You can play with the page here (you might have to allow the browser to run the embedded script). Every reload will give you a completely new, randomly generated look.

Yes, there are a few horrible-looking duds, but a surprising fraction of the generated styles don’t look half bad, a few even look very good.

One can imagine stretching this in many interesting directions. For example, you could perform sentiment/mood analysis on the body text, and match it with a typeface that has similar characteristics. Of course, there are also the million other CSS variables that I didn’t change.

I’ve argued previously that a lot of what today’s creative professionals do is variation within a theme, and that can be easily replaced with an algorithmic system that incorporates enough randomness to make the output look organic. This is another small example along those lines.

Here are some screenshots, starting with the vanilla unstyled page. I used the page from the CSS Zen Garden.

Hardware

• We’re in a transitory period now, which is why there are still people who like to use machines with tons of local storage. That is about to become irrelevant. The datacenter is your compute and storage. The device in your hand (be it a phone, a tablet, or a laptop) is your modern “dumb terminal” with pretty graphics that acts as a conduit to all that data. Local, on-device storage is just a cache.
• Find a computing setup that does not harm or injure your body.

Data

• Treat service providers like banks. When you store your money in a bank rather than under your mattress, you are trusting them, while buying yourself the convenience and flexibility of not lugging around sacks of cash. But there are also a number of safeguards and incentives in place for the banks to keep your money safe and liquid. It is the same with cloud services holding your data. Also, they know this stuff much better than you. Corollary: run from providers that make it hard to get your data out.
• Data longevity is a hard problem, but you can improve your chances by using plain text as much as possible (preferably in a long-term-friendly encoding like ASCII or UTF-8), and the most-used file formats for images (I think that’s JPEG).
• Keep one local backup of all your cloud data for the ultimate apocalypse scenario.

Software

• Over the long run, it is your data that matters, not the software. That’s why this goes last. That wedding photograph from 1999 is more important than the image editor you used to touch it up. Pick one for each of your common activities (writing, image management, browsing etc.) and stick with it. Don’t listen to the lifehackers and productivity gurus–moving to the latest shiny editor will not make your prose better.
• Corollary of the above: never use software that locks you into proprietary formats, or if you must, make sure to export your files out to a more portable format. The chance that you will be able to run the same hardware/software/version snowflake in a decade to decode your data is close to zero.

The opening question to everyone:

A 22-year old college student has been accepted to a Ph.D. program in a technical field. He’s academically talented, and he’s always enjoyed school and his subject matter. His acceptance is accompanied with 5-years of guaranteed funding. He doesn’t have any job offers but suspects he could get a decent job as a software developer. He’s not sure what to do. What advice would you give him, or what questions might you suggest he ask himself as he goes about making the decision?

Kudos to Tim for creating a great resource on this topic.

I want to start by saying that I greatly admire Berkun’s work in general, and have read almost all his books (I think everything but Confessions of a Public Speaker). That said, I don’t think YWP is his best to date.

YWP is an excellent account of, well, Berkun’s time at Automattic, and its culture and techniques viewed through the eyes of a “traditional” project manager. There is a disconnect in the way the book was pitched and what it actually contains. It was pitched partly as a book about remote work being the future of work, and partly as a book about how to make work more fun. It is tangentially about both those things, but the primary thrust of the book is the personal narrative of the author working and shipping at Automattic.

What was missing was any discussion of how to take Automattic’s specific culture and tools and ways of working, and extrapolate them to other settings, such as a new company, or an existing traditional one. (He later wrote about this on his blog.)

Also, for a book that wants to sell the idea of a distributed company, a significant fraction of it ends up detailing the meet-ups where teammates physically met and worked together for a few days. All team-members would come to a given city for a week, camp out in a large hotel room or apartment, and basically code, eat and drink (and maybe sleep a little) the whole time. That might be exciting and fun, but it sounds like a horrible way to work (even for one week) for someone older or with a family.

So it was a great account of how Automattic has had success with the distributed team model, but left me wanting much more. Berkun particularly shines when talking about team dynamics, leadership and project management, and I did learn quite a few new ideas and techniques from that.

Some interesting questions that I think a book on distributed teams and remote working should address:

• how can an existing company either accommodate or make the transition to being distributed? This is the big one. Because existing companies dwarf new ones. If one wants to change the way work works, one has to tackle this thorny problem.
• how can distributed work scale? By now it is not controversial to claim that distributed teams and remote work is effective for small to medium companies. But how can we scale it to larger teams and companies, going from hundreds to thousands of employees?
• learn from the Linux kernel. To me, the most successful large distributed team ever is the one that builds the Linux kernel. The artifact is complex and is now the substrate for the modern internet, and the team building it has thousands of contributors. And it has been delivering new releases and features and bugfixes steadily for more than a decade. And it has been doing this with almost no overarching oversight, other than Linus and the subsystem lieutenants weighing in on technical matters. The amount of physical proximity with other kernel hackers is small compared even to the case of Automattic–most kernel hackers will meet their fellow hackers once a year at a Linux conference. To those living on LKML and slinging patches around, this is a completely natural way to work and has become second nature. To anyone outside, it seems like an impossible feat, especially when it seems to be so hard to ship any software even with physically co-located teams and tons of oversight. Anyone who wants to understand distributed teams has to take a long, hard look at the Linux kernel community and try and learn from what they are doing.

But all the same, I do look forward to Berkun’s future work.

The year is 2045, and my grandchildren (as yet unborn) are exploring the attic of my house (as yet unbought). They find a letter dated 1995 and a CD-ROM (compact disk). The letter claims that the disk contains a document that provides the key to obtaining my fortune (as yet unearned). My grandchildren are understandably excited, but they have never seen a CD before—except in old movies—and even if they can somehow find a suitable disk drive, how will they run the software necessary to interpret the information on the disk? How can they read my obsolete digital document?

From there the paper goes down a twisty maze of reasoning to lay out exactly how difficult it is to preserve digital documents for the long term, i.e. hundreds or even thousands of years. We have stone and paper documents dating back thousands of years, but it is extremely unlikely that today’s digital documents will make it that far into the future.

There are problems at every level of abstraction. The physical media on which the bits are stored will decay. The documents have funky encodings and metadata that can only be parsed and displayed by the programs that were used to author them. Those programs will only run on certain OSs. Those OSs will only run on certain hardware. If each of those hurdles are overcome, we have some hope of recovering the old document.

That’s like raising an already small probability to the fifth power.

Like I said, it’s depressing.

And the biggest issue is that digital preservation is an active, ongoing task. I could print something on archival paper, store it in a safe deposit box, and be reasonably sure it will be readable in a century. Trying to do the same for a digital document would require fiddling (copying to newer media, maybe changing formats and encodings) every 2-5 years.

Maybe the Library of Congress will go to all this trouble to preserve digital documents it deems significant to the record of our civilization and times. What hope do I as an individual have to carry forward my digital life to the point where I could hand it over to a grown child, or even grandchildren?

I learned a small lesson early in my digital life and have been putting as much as possible in plain text files. But “plain text” can be one of many encodings, and you need to know it independently of the file.

On the table where I write this, I have a black and white photograph of my mother when she was about twelve years old, with my then young grandparents. Will I be able to give my twenty-year old child photographs of the vacation we went on when he was a toddler?

The overall message has many similarities to Ericsson et al’s famous paper about deliberate practice (of which I wrote a summary), in that “innate talent” as something that one is born with is a useless concept used to mystify the systematic, methodical and disciplined practice of otherwise mundane habits. Hence the title of the paper.

The author concludes that:

• The levels of achievement (juniors, seniors, nationals, Olympics) are discrete rather than continuous, and the practices and attitudes within each are different enough to make them disjoint parallel worlds.
• The jump from one level to the next one comes not from quantitative improvements (i.e. doing more of the same thing, or doing it faster), but from qualitative shifts such as new and different techniques, attitudes and practices.
• Other than that, there is no secret to achieving excellence, and indeed, the primary psychological barrier to achieving it is to get over the sheer mundanity of it.
• “Talent” is indistinguishable from its effects. We only call someone “talented” if their achievements already make it obvious. The cult of talent is actually harmful because it obscures the true, achievable (but mundane) path to excellence.

I read a lot, because I don’t believe minimalism is a great intellectual strategy. I have thoughts and opinions about what I’ve read, and sometimes about other stuff that springs unprompted in my head. I want to post those on the great wide internet for all to see, because, well, that’s what it’s for, isn’t it?

I also want to keep a high bar for posts on this blog. That means I polish it (as much as one can polish a blog post) and try to be coherent and have a complete thought. That takes time, and often the latency between initial spark and hitting “publish” is weeks or months.

However, in my own narcissistic way I do believe all that raw material (sometimes just a list of links, sometimes half-baked thoughts) might be valuable and interesting in and of itself.

And that’s where the newsletter comes in. I want it to be a more intimate channel to share thoughts that that are not yet ready for the harsh glare of the public web, and maybe even start a conversation and learn something.

If you subscribe, you can expect around 1-2 emails a week, on much the same set of topics that I write about on this blog: programming, technology, modern work, and occasional side-trips into culture and criticism.

I invite you to subscribe.

I’d love to hear of any other good references on the topic.

• “Conventional wisdom … holds that distributed software development is riskier and more challenging than collocated development… [We compare] the post-release failures of components that were developed in a distributed fashion with those that were developed by collocated teams. We found a negligible difference in failures. This difference becomes even less significant when controlling for the number of developers working on a binary. Furthermore, we also found that component characteristics (such as code churn, complexity, dependency information, and test code coverage) differ very little between distributed and collocated components.” The paper goes on to detail some of the practices that contributed to making this difference small. Among other things, “developers made heavy use of synchronous communication daily. Employees took on the responsibility of staying at work late or arriving early for a status conference call on a rotating basis, changing the site that needed to keep odd hours every week. Keeping in close and frequent contact increases the level of awareness and the feeling of ‘teamness’. This also helps to convey status and resolve issues quickly before they escalate. In addition, Engineers also regularly traveled between remote sites during development for important meetings.” ¹
• A decade ago, distance mattered, a lot. In spite of that, since then, distributed work has only grown, fueled by both economic and other considerations (e.g. the talent may not be where you want them to be). A range of technologies has made this easier: shared documents and calendars, instant messaging, video conferencing over the net; but also a realization that distributed, virtual leadership is a distinct and valuable skill. ²
• Distributed development was “considered harmful”, but recent rechecking of those results has revealed that “the effect size of the differences seen in the collocated and distributed software was so small that it need not concern industrial practitioners. Our conclusion is that … distributed development is not considered harmful.” ³
• Among the behaviors that have the biggest impact on effectively working remotely: communicating changes and status to downstream dependencies, clear communication of who owns what, responsiveness to emails and IMs, and co-operation (which means actually helping and unblocking someone). ⁴

(All emphasis mine.)

You must have seen the warning a thousand times: Too few young people study scientific or technical subjects, businesses can’t find enough workers in those fields, and the country’s competitive edge is threatened.

It pretty much doesn’t matter what country you’re talking about—the United States is facing this crisis, as is Japan, the United Kingdom, Australia, China,Brazil, South Africa, Singapore, India…the list goes on. In many of these countries, the predicted shortfall of STEM (short for science, technology, engineering, and mathematics) workers is supposed to number in the hundreds of thousands or even the millions…

And yet, alongside such dire projections, you’ll also find reports suggesting just the opposite—that there are more STEM workers than suitable jobs. One study found, for example, that wages for U.S. workers in computer and math fields have largely stagnated since 2000. Even as the Great Recession slowly recedes, STEM workers at every stage of the career pipeline, from freshly minted grads to mid- and late-career Ph.D.s, still struggle to find employment as many companies, including Boeing, IBM, and Symantec, continue to lay off thousands of STEM workers.

The flaw is in the assumption that every STEM degree holder is a qualified candidate for a STEM job. Carried through to its logical conclusion this implies that simply presenting a STEM degree should be enough to get one placed in a STEM job, which is absurd.

This becomes readily apparent if one has ever been on the evaluation side – grading students, or conducting interviews for jobs. I’ve been a TA for a number of CS classes while in grad school, and I’ve conducted many interviews for software engineer positions. Just from my narrow anecdotal window, it is amazing how many CS students just want to figure out the bare minimum to pass the class; and how many grads do not have a decent grasp of elementary algorithms and data structures, and are not comfortable with code.

When CEOs of tech companies complain that there are not enough STEM workers to fill their open positions, what they’re really saying is that it is very hard to find the right calibre candidate, the glut of credentialed STEM graduates notwithstanding.

Update: HN has some great comments, many from people in the trenches.

I’ve lost count of how many national parks and monuments I’ve seen over more than a decade, but at each and every one of them, without fail, the rangers were just flat out brimming with pride and joy.

They wanted me to see, truly see, not just the visual scenic beauty, but the deeper connections in the ecosystem, and the geological history that made the place what it is.

Some shared personal stories about how and why they became rangers.

Some quoted John Muir and Thoreau extensively.

Some spoke of the simple grandeur and solitude of living miles from civilization for months.

All of them spoke of the delicate preciousness of their park, looking over it with loving watchfulness.

All of them were humble, emphasizing how much they don’t understand about nature, and how little they do.

Surely they have challenges and frustrations in their line of work. It’s just that I’ve never seen it as a visitor.

There is so much ink spilt writing about how to be happy at work. But what I see in them makes the word “happy” seem flat. It must be what one feels when one is tasked with looking after something precious and lovely, yet delicate and needing protection.

We Are All Remote Workers http://t.co/KuBQgfMkWg < my talk at #romagnacamp, also inspired by the works by @vivekhaldar @joelgascoigne

— Nicola Iarocci (@nicolaiarocci) September 6, 2013

But even though the web is the delivery mechanism for long-form pieces, I almost never actually read them in the browser. It just doesn’t feel right. First of all, I’m not in the frame of mind to read long-form when browsing. Secondly, no matter how clean and tasteful the page design, I prefer something even more stripped down, like Readability.

When I come by a long-form piece I want to read, I add it to my Pocket queue, and usually read it on my tablet, laid back, without any other distractions on the screen.

So the Web has become an excellent delivery mechanism for long-form content, but its actual enjoyment happens elsewhere.

Who is a remote worker? What qualifies a piece of work as remote, as opposed to situated? The answer is more subtle than it appears at first glance. The first criteria that comes to mind is: physical proximity to collaborators. But access rapidly drops off. It works well if you can turn your chair around and talk to a colleague, but if they’re on another floor or in another building, then they might as well be in another city. Add to that the cult of “respect the headphones”, especially for programmers. More often than not programmers in the same room will have giant ear muffler headphones on and communicate with each other via chat and email.

No system is an island, and modern code will typically build upon a number of other services or libraries. If you’re writing code against a number of other services and libraries, the chances that the people you need to collaborate with for all of them are physically close to you are infinitesimally small.

Hence, a good fraction of your work is likely “remote”. So whether you’re a remote worker or not, it will help you if you build skills for effectively working remotely.

The most important mental shift is towards working asynchronously. This is hard, because it requires some thought and planning, and is a lot more proactive than synchronous work. Working synchronously by default is lazy and ineffective.

A corollary of working asynchronously is that communication shifts away from verbal and towards the written word. Writing coherently is harder than informal conversation. Polish your writing skills.

And when you finally do meet those far flung colleagues, remember not to just talk shop! Build the trust and rapport that will enable you to work through the ether.

P.S. I’m looking forward to two books about remote working scheduled to release this fall: Scott Berkun’s “The Year Without Pants”, and 37 Signals’ “Remote”.

Before most computer science graduates ever walk across a stage to get their diplomas, they’re set for life. This is especially true in 2013, which will be the first year in which most companies pay top engineering graduates in Silicon Valley $100,000 or more per year in salary. For the companies, for Redfin, the engineers are worth every penny. And for the engineers, the money is nice to have. But how many engineers hired from Stanford or Berkeley in the past year will ever feel the savage need to make something happen, to bust out of the matrix, to push the limits of their abilities? The problem is that the young engineers earning that much become well-fed farm animals at the very moment in their lives when they should be running like wild horses.

(Emphasis mine.)

The piece plays into several stereotypes:

• The Startup is the apogee of all possible corporate forms.
• The Startup is the only place where an engineer can do hard, challenging work.
• Above benefits are great enough that one should take on significant risk, financial and personal.
• If you are young, you have nothing to lose, so roll the dice with a (preferably my) startup!

This is the Overarching Valley Myth. But reality is much more varied, much more complex, and not as disappointing.

If hard technical problems in the real world are what you seek–and that is indeed what you should seek if you’re a fresh CS grad–then a startup is possibly the worst place for you. Startups are about product definition and finding a business model. They are only tangentially about technology. Given all the risk that a startup takes on the business side, the last thing it needs is to take take on more risk on the technology side. Corollary: startups prefer middle-of-the-road, mature technology.

On the other hand, large tech behemoths are good places to dwell on hard technical problems. They operate at a scale where such problems actually manifest themselves. They also have enough resources that they can comfortably push the envelope on new and risky fundamental technologies without worrying about whether they have enough money to make payroll next month. That’s why they make a much better training ground than startups.

For example, there is no doubt that you will learn much more about planet-scale distributed systems at places like Amazon, Facebook, Twitter and Google, than any startup under the Sun.

Kelman is also downplaying the significant risk one takes on with an early stage startup. Financially, a big chunk of compensation at a startup is equity, which of course will be worthless if the startup fails. The flip side is that you could hit the jackpot and become wildly rich. But you know the odds. Also know that if successful, rewards go disproportionately to investors and founders. So why not make a startup instead of joining one?

But there are personal costs too. The perfect counterpoint to Kelman is Alex Payne’s Letter To A Young Programmer Considering A Startup, which takes a harrowing personal look at the toll of the Valley Myth.

I’ve seen firsthand the damage that startups can do to relationships. I’ve watched marriages and friendships fall apart, seen children and partners pushed aside, and failed those in my life in all kinds of ways when work came to the fore. I’ve listened as people who are the very picture of startup success – visible in the press and social media, headlining conferences, forever founding and exiting – have confided their utter loneliness despite being seemingly at the social center of the entrepreneurial community.

There are two sides to every story. The truth is probably somewhere in the middle.

Kelman is also reinforcing the Valley age stereotype, what with all the imagery and metaphors around young, wild horses. Yes, when you’re fresh out of school, with no family and no mortgage, you have nothing to lose. But I think he’s completely missing the other side of the age/risk curve, which is engineers who have spent a decade or so at a large company, and built a comfortable financial cushion. They’ve built up an appetite for risk by then, and they’re experienced. They won’t run around like wild horses. They don’t need to. They know exactly how to build your system right the very first time, with a minimum of wildness.

Last but not least, Kelman has a massive bias: he’s complaining because he runs a startup, and startups are finding it hard to hire and retain engineers because all these other large companies in the Valley have soaked them up with pumped-up salaries and cushy perks.

Buried deep in Kelman’s piece is a sentiment I agree with: don’t get comfortable. Keep stretching yourself. Being showered with perks doesn’t mean you cannot keep seeking out hard problems. Even while burping from your gourmet meal during a mid-afternoon massage.

[Full Disclosure of Personal Bias: I happen to be happily ensconced in a large tech company. That’s partly why Kelman’s piece pricked me. Whether it has made me comfortable and soft time will tell.]

There are many examples:

• As Burton Malkiel points out in his book “Random Walk”, asset allocation accounts for > 90% of the returns of an investment portfolio.
• Google is famous for its 70/20/10 allocation: 70% on the core business, 20% on related businesses, and 10% for wild, blue-sky projects.
• The projects you personally deal with will make progress in proportion to the fraction of your time you devote to them.

That seemingly simple issue is actually the tip of the iceberg for a bundle of intertwined complex ones. It usually encompasses some or all of the following:

• Maintenance is boring, bad, and uninteresting.
• I only want to work on new and interesting stuff. I want sexy projects.
• I want to work on projects that will advance my career, and maintenance projects are not it.

Maintenance is boring, bad, and uninteresting.

No, maintenance is important.

Look around you, at the modern civilization you live in. The primary activity of civilization is maintenance. A new building or a new road pales in comparison to an existing city. They also have to connect with the existing city. Engineers who work with things other than software (mechanical, civil etc) seem to have a deeper understanding of the primacy of building on existing infrastructures.

All this is to say that maintenance doesn’t get its due. Software now is just as much a part of the infrastructure of civilization as roads, bridges and cities. As time goes on, its maintenance will get more important, not less.

I only want to work on new and interesting stuff. I want sexy projects.

Innovation is often misunderstood. We have a tendency to mythologize innovation as a “Eureka!” story, but zoomed in, innovation looks like a long thread of small, incremental steps.

In a paper delving into the challenges that new grads faced in their first software engineering jobs, the authors found that:

Our study reveals that new developers find themselves in situations that differ considerably from the university [software engineering] class… We see new developers joining large, preexisting teams of software developers as the most inexperienced member, and spending their first several months resolving bugs that predate their employment, with little access to easily consumable documentation. Many of the problems they have are not due to lack of experience in programming, design or debugging. In fact, all of our study subjects said their university preparation in these areas was more than adequate. The problems instead centered around the particular social conditions of a new software job. Instead of a greenfield project, a more valuable experience would provide students a large pre-existing codebase to which they must fix bugs (injected or real) and write additional features.

Of course, this is true not just of fresh grads, but of experienced engineers as well. The vast majority of their time is spent extending existing systems.

Truly green-field projects are rare. And they are risky, which helps make them rarer. But the fact is that you would make a crappy green-field project if you had no experience maintaining a real system. Because–guess what–very soon your green-field system won’t be green-field any more and you will have to maintain and extend it.

The shift away from clients towards large cloud systems only increases the slant towards incremental changes, because an online service cannot be replaced in one shot. It has to be changed piecemeal, akin to changing parts on a plane while it is flying.

How will you tell a “sexy” project from one that’s not? All the things you think will indicate a “sexy” project–management excitement, newness–have no correlation with the ultimate success and impact of a project. When you say you want to work on a sexy project what you’re really saying is that you want to be on a successful, celebrated project. And you can’t spot those a priori.

I want to work on projects that will advance my career, and maintenance projects are not it.

So far I’ve been debunking. But this part of the concern is actually valid.

You want to be judged by impact of the projects you worked on. Whether they are “maintenance” projects or not is immaterial. Instead of specific projects to work on, look for a place that has a performance process that is objective, gauges impact, peer-based and not reliant on just your manager.

Previous related post: Shitty legacy maintenance.

If the cost of memory safety bugs in C/C++ codes is significant, and if solutions are available, why aren’t we using them in production systems?

He offers several possible reasons:

The cost of enforcement-related slowdowns is greater than the cost of vulnerabilities.

The cost due to slowdown is not greater than the cost of vulnerabilities, but people act like it is because the performance costs are up-front whereas security costs are down the road.

Memory safety tools are not ready for prime time for other reasons, like maybe they crash a lot or raise false alarms.

Plain old inertia: unsafety was good enough 40 years ago and it’s good enough now.

I’ve been thinking about this too. This was a hot research topic not so long ago, but seems to have faded away. I think there are a few additional factors at play here:

• The cost of memory unsafety is much higher for client software than for server software. As our world moves towards thin and light clients (browsers, apps) that access heavy backend services, the urgency of the memory safety problem has diminished. Paying a small overhead (say, 5-10%) for memory safety on a laptop or desktop does not sound like much, but adding 5% CPU to an entire datacenter is material. At the other extreme, on phones and tablets, conserving battery is important and any drain on that needs a strong justification.
• In tandem, other security problems–cross-site scripting, SQL injection–have become important to tackle.
• Tools like address sanitizer have done a pretty good job of weeding out the vast majority of memory safety bugs early in development (with unit tests), taking the overhead from runtime to compile/build/test-time. The tradeoff is giving up soundness, i.e. you are not guaranteed to catch every bug.

All these combined haven’t “solved” the C/C++ memory safety problem, but they have brought it down to the level where it is manageable, and its current cost is judged to be comparatively low.

I took color printouts of his cartoons and stuck them on my cubicle walls. I felt somewhat guilty about him not making anything off that, but then, the real prints were kind of expensive. I bought his books to redeem myself, but they were worth it on their own.

So when he came out with something I had been looking for for a while, a unique decal for my laptop, at a price I didn’t have to think too hard about, I jumped at it. The landscape looks barren when the fifty laptops around the conference room all sport combinations of the same ten stickers.

In nearly a decade, during which time he has given me something cool and new to look at nearly every day, he has made less than a hundred dollars from me. That’s a ridiculously small amount of money given the consistent pleasure I have derived from his work over a long period of time. But that’s emblematic of the new economy.

As MacLeod himself says, “Have something to say, have something to sell”. He lives that. And I thank him.

• comments that were initially down-voted were corrected, so that their final ratings were not statistically different from that of the control group.
• comments that were initially up-voted saw more benefits: their final ratings were on average 25% higher than the control group.

This reminded me of a couple of papers²³ by Duncan Watts and Matthew Salganik⁴ a few years ago that look at a similar question in the context of songs: how do social signals (like ratings and prior downloads) affect the success of a song in an online market?

They used a slightly different metric, the Gini coefficient, to measure inequality in the resulting market. In the control case, people downloaded songs without being shown how often they had been downloaded in the past. In the social influence case, prior downloads were presented. Not surprisingly, “influence worlds exhibit greater inequality—meaning popular songs are more popular and unpopular songs are less popular—than the world in which individuals make decisions independently.”

But more interestingly, Watts and Salganik found that social signals increased the unpredictability of success. They created “parallel worlds” of users for each case. The variability of the success of a given song across worlds was much greater in the social influence worlds than in the independent ones.

And their final insight was the relationship between “quality” and success. They measured quality of a song by its success in the independent worlds. They report that “in general, the best songs never do very badly, and the worst songs never do extremely well, but almost any other result is possible.”

Ever since James Surowiecki published “The Wisdom of Crowds”, I’ve often heard the glib “rule” that “many minds are smarter than one.” But as Surowiecki constantly stresses in the book, crowds are wise only if each individual judgment is uncorrelated with the others. When the judgements are correlated, you don’t have a wise crowd, just a stupid herd.

With 1.1, you could literally read off what was on the wire and make sense of it, because it was just plain text. Now you’ll see a stream of opaque bytes.

To me, this marks the end of an era. The early web was designed to be legible to humans, at the expense of some machine resources. Plain text was surely not a compact representation, but it was the simplest one. Chatter between machines and humans was as important as chatter among machines. The web has now reached the maturity and scale that the cost equation has flipped.

Pictures are from Arstechnica’s review of the Moto X.

There are problems with pegging your aesthetic to technology though. It moves too fast. For example, the earliest NA pieces were heavily influenced by digitization and quantization, the fundamental process by which physically continuous objects are ingested into the discrete digital universe. The pixel became a common NA artistic element.

Of course, with the spread of high-resolution retina displays, the pixel is now a quaint artifact of the previous decade.

The NA movement has adapted though. It’s current thrust seems to be moving away from literal digital graphical views, and instead interpreting the ways in which machines read our world for their own purposes, recognizing streets, objects, faces and barcodes.

Retro New Aesthetic

In an ironic twist, the NA has taken a retro turn, trying to bring crafted old-school analog warmth to our cold mass-manufactured digital devices. They are warm organic shells for cold steel.

The prime example is the USB Typewriter. At nearly a thousand dollars each, they are priced like pieces from an art gallery too.

Will your typewritten words feel more organic if their conduit is one of these wooden keyboards, “made from a single piece of wood to preserve wood grain across shell and keys”?

Or consider the smartphone, our umbilical to the digital matrix, made out of bamboo:

That is going to fit into one’s cabin in the woods so much better than the glass-and-steel iPhone.

What we make, makes us

Where does this go next? The most promising direction seems to be how our physical world is carved, sometimes literally, by our digital understructure. Kevin Slavin does a wonderful job of explaining that in his TED talk:

But algorithms can carve more than just our physical world. They can shape us, as humans.

If you carry this line of thought to it’s logical conclusion, the next step is inverting the machine-human relationship to where people become parameters for algorithms. Amazon’s Mechanical Turk is a crude first version of that vision. There are plenty of scenarios where human brain power is cheaper than burning CPU. RPC is the abbreviation for “remote procedure call”, but it could also mean “remote person call.”

Our algorithms shape us, and they shape our culture. Would you really ever have seen that niche documentary or gotten into that indie band if Netflix or Pandora had not recommended them to you? Would you choose them? Is the machine aiding or subverting your free will? If recommendations are perfect and encompass everything, not just movies and songs, what does free will even mean?

Those are the questions the NA has to deal with. It has to define what art is in a world where humans make machines and machines make humans and the digital and physical have no clear boundary.

Some points that jumped out at me:

• extrinsic properties like library availability and social factors are much more important than intrinsic factors like language features
• C++ was by far the hardest language to master. Java, JavaScript and C# were in the middle. Python and Ruby were the easiest.
• There was almost no variation with age in the number of languages one is proficient in. Good data to use against ageism.
• There is a high correlation between enjoying a language and its expressivity.
• Static typing still has a massive PR problem. Only ⅓ of developers find static types valuable, compared with ⅔ who find unit-testing valuable. “This suggests that today’s type systems may err too much on the side of catching bad programs rather than enabling flexible development styles.”
• Performance was ranked the 2nd most important feature (after libraries), but specific language features that help performance ranked much lower, which shows a “gap between the importance of performance and the language features used to achieve it today.”
• Advice for language designers: “Since languages grow niche-by-niche, designers should focus their marketing and library-creation efforts on particular communities. Growth comes by expanding to new domains.” Examples are numpy for scientific programming in Python, and Ruby on Rails for webapps.

Previous related posts:

Whither programming language research?

Innovator’s dilemma in programming languages

How to make your new programming language successful

And then came the network, and everybody and everything else with it, and it was like that software had broken out of the playpen and could now run around the house, the neighborhood, the entire city.

It was only logical that the element whose entire raison d’etre was connectivity to this new computational universe, the browser, ended up subsuming and becoming the container for all our applications. We moved from apps to webapps, but by now the web prefix already seems redundant. The majority of people can function just fine in both the office and the home working completely in the browser.

We’re seeing a similar battle between two types of apps in the mobile world. Native vs webapps. Which is the better fit for mobile? Which should you build first?

What’s different is that connectivity is now the default. Whether your app is contained in a browser or not is less important. What’s more important is the networked computational universe they have access to. Most apps are pretty data frontends to vast remote backends that do most of the heavy lifting.

In a somewhat ironic twist, webapps are getting better and better at working offline, for those rare times when you don’t have a connection. Some have argued that apps should be designed to function offline first, treating the network as just another progressive enhancement.

We’re in the same place with mobile now that we were when GMail was first released on the desktop. The browser showed us a glimmer of what was possible, but it had yet to take on the abstractions and power that would make it a decent platform for real apps. The browsers on mobile devices still don’t have full access to the raw hardware (gyro, camera, various other sensors), but it’s just a matter of time before those differences are erased.

That’s why I think native mobile apps are just a stopgap until mobile browsers catch up.

That was when I began to sour on the term “knowledge worker.”

There is an implicit connotation that knowledge work is higher-skill and higher-status. Matt Crawford stuck a nail into that notion in his book “Shop Class as Soul Craft”, in which he delved into both why “knowledge work” can be dull and soul-crushing, and working with your hands on physical objects can be extremely cognitively challenging.

As a programmer, most of what I do is challenging and interesting, but once in a while I get the creepy feeling that all I’m doing at my computer is pulling levers. Think about how much code you can churn out in a modern IDE by just auto-completing, then think about the frightfully small distance to go before even than can be automated. Dan Benjamin often mentions on his podcast how easily replaceable Java developers are. “When you’re gone, the floor will part and another Java dev will appear, shrink-wrapped with his chair. They’ll wheel him to your cubicle, take off the shrink-wrap, and business will continue as normal.”

The culture of Silicon Valley can be traced back to Robert Noyce, and he was dead-set against unions:

As the work force grew at Intel, and the profits soared, labor unions …made several attempts to organize Intel. Noyce made it known, albeit quietly, that he regarded unionization as a death threat to Intel, and to the semiconductor industry generally. Labor-management battles were part of the ancient terrain of the East. If Intel were divided into workers and bosses, with the implication that each side had to squeeze its money out of the hides of the other, the enterprise would be finished. Motivation would no longer be internal; it would be objectified in the deadly form of work rules and grievance procedures. The one time it came down to a vote, the union lost out by the considerable margin of four to one. Intel’s employees agreed with Noyce. Unions were part of the dead hand of the past.

This ethic carries on today. The idea of unionizing tech workers is still considered oxymoronic. And that is a good sign because the hankering for unionization is a leading indicator that your field is going to get eaten by automation. It means that the only bargaining chip you have left is control over the aggregate supply of labor, not skills or creativity.

For another example, take law. What was once thought of as purely human and beyond the touch of machines is crumbling to automation. Robot lawyers are creeping into the bottom end (routine contract generation, search and discovery) and gradually climbing up the value chain.

Claims made about the impact of automation on the law are not entirely speculative, nor are they new. E-discovery, which applies modern search technologies to help manage the massive amounts of data in litigation, has already seen significant coverage both within and beyond the legal industry in the past decade… However, what may be more novel is that automation is moving increasingly beyond incremental improvements to tools used by lawyers in the “back office.” Automation more and more touches the actual work product received by clients, as well as “front office” interfaces that the public uses to access legal services and the legal system at large.

This isn’t manufacturing and factory workers the author is talking about.

Clearly, the moat of security around being a “knowledge worker” is drying up fast.

At this point, if you are a knowledge worker, you should be wondering whether your job can be saved, and how you’ll know.

Venkat Rao has a deep meditation on this very topic. Go read the whole thing. The tl;dr is that while we worry about saving the sexy a.k.a creative work from automation, the real opportunity for human gainful employment lies in dull, unsexy, schlep work around the fringes of what machines do and that are not worth automating. In other words, exploiting the arbitrage when your brain is cheaper than CPU time. The best example of this is humans doing image processing on Mechanical Turk. The essay is an excellent deconstruction of the platitude that “computers can’t take over creative work.” If you dig deep down, a lot of creative work isn’t creative at all.

A lot of what people think of as creativity is variation within a theme. Computers can actually be pretty good at it. Consider the following three computer-generated pieces of art:

All that intricacy and variation was generated by the following code and a sprinkling of randomness:

startshape S
background { b -.5 sat 1}
rule MYSQUARE {
    SQUARE { b -1 }
    SQUARE { s 0.5 }
}
rule MYCIRCLE {
    CIRCLE { b -1 }
    CIRCLE { s 0.5 }
}
rule S {
    X1 { x -.5 y -.5 }
    X1 { x -.5 y .5  }
    X1 { x .5  y -.5 }
    X1 { x .5 y  .5  }
}
rule X1 .13  { MYCIRCLE { b 1} }
rule X1 .13  {
    MYSQUARE{ b 1}
}
rule X1 .74  { S { s .5 b .3}  }

A good fraction of human creativity follows the same pattern of generative expansion from a small kernel. Is it that hard to imagine a program that generates pleasing web designs and puts web designers out of business? You would need a generative component–one that would generate a large number of designs given a small spec–and a filtering component that would whittle them down to a small number of top designs that a human could then select from based on whimsy. The first component is easy to write today. The second component becomes possible if one feeds a large number of “good” web designs to a machine-learning program.

So creativity isn’t going to save knowledge workers. That moat is looking thinner and dryer.

A useful direction to look at is ambiguity. Dealing with ambiguity is the step before creativity. The creative process comes after beating down the ambiguity of what to create and the constraints within which it must exist. In a sense it is meta-creativity. I’m still stumbling through this idea, but my gut tells me that chasing ambiguity is likely to be an effective hedge against machines eating your job. Stay tuned.

• Lambdas with generic types. ML can eat it’s heart out. This will become possible in C++:

  auto lambda = [](auto x, auto y) {return x + y;}

• Extending auto to infer the return type of functions. Though a part of me still wants to see them just for documentation.

Nobody tells you how to handle email in a large modern organization. You learn through pain, osmosis and experimentation and end up with your own unique snowflake of subscriptions, filters and settings. Picking an email platform and setup is like voting–there’s no candidate or party that exactly matches your views, but you just pick the one that least offends your humanity and decency.

Working on the assumption that what I’ve ended up with might be useful to others, here’s how I do email. I’m only dealing with incoming messages here. There are many other fine articles on the web delving into the art of crafting an elegant, effective electronic message. Also, this is for work email. Personal email is much easier.

First of all, don’t even call all your email “email.” The term has hidden connotations and expectations of responsivity. The truth is that only a tiny fraction of the messages in your inbox are “email” in the traditional (1990s) sense.

The golden rule

Before we get to nuts and bolts we have to lay down the overarching principle. And that is: your inbox and how you deal with it must reflect your professional goals and priorities. Which is to say that you have to categorize your email by where it falls in your professional pecking order. Usually, it goes something like this, from most important to least:

• messages where I am explicitly on the “To:” line. This means it impacts my day-to-day work, helps me make progress on what I’m doing right now, or I’m expected to respond in some way. The response could be simply chiming in with an opinion that others will take into consideration, or a straight answer to a straight question, a clear decision, or… something else. If it’s something else, then email might help you get to the bottom of it, but you will usually need something stronger, like IM, video chat, or even a face-to-face to truly resolve it.
• messages to my team’s mailing list. The hub of most modern teams’ day-to-day heartbeat is their mailing list. I’m not going to be explicitly on the “To:” line here, but the expectation is that every team member peruses this at least a couple of times a day to stay abreast of what’s going on in their immediate professional environment. Is the build broken? Did someone send out a new design for review? Look at this massive change coming down the pipe!
• messages to the mailing lists of teams that my team interacts with. These might be teams whose work you build upon, or those who build on yours, company-wide infrastructure, ops etc. Occasional perusal (a couple of times a week) is all you need here.
• everything else. This is just organizational atmosphere. Usually not even technical. And if something really big happens, you’ll probably hear about it before reading the email.

Rules of the road

There are a couple of other ground rules if you are to stay sane while dealing with the hose of modern email:

• email is a terrible place for a todo list. I really don’t get how one can tack on GTD to email. An email is not actionable. Once you read it and understand it, you might end up with a clear action, but your inbox is not the place to shove it.
• email is not the place where other people get to create work for you to do. I suspect this is the primary reason everyone is so stressed out by email. If every message is a chunk of work falling on you, no wonder it drives you mad. It took me a long time to realize this. The work you are supposed to do is discussed, clarified and assigned via other channels, usually chats with your manager. Email is for carrying out those predetermined responsibilities. Random people do not get to put tasks on your todo-list by simply sending you an email. You must hold a steady stance with this. If they do, you have to push back, politely, and clarify the situation through other channels. (I suppose an exception is if you are in some sort of customer-support/CRM role.)

Implementation

I use GMail, so everything below is going to be in terms of that. It’s pretty simple really. I have filters for each of the organizational circles described above. It’s important to be fluent with GMail’s handy keyboard shortcuts so that you can switch between views quickly.

For example, everything to “my-team-list@foo.com” gets labeled “myteam”. I can quickly switch to that view by pressing “gl” followed by the first few letters of “myteam.”

The unread counts mean nothing to me. Everything is a stream. Some messages go by unread because I get what I need from the subject line. The only exception is below.

The view I use for the innermost circle (i.e. messages explicitly to me) is the one where I spend the majority of my inbox time. The filter is use for that is “to:me is:unread newer_than:7d”, which gives me all messages explicitly to me sent in the last week that are unread. If I want blinders to just zoom in, I can drill down to just “newer_than:1d” for just today’s messages. Or I can use a larger number of days to zoom out.

Go do email. Don’t stress.

I recently turned thirty-five. I never thought I’d live to see the day. By that, I don’t mean that I expected to die by now, but that even in my twenties, the number 35 seemed too far to fathom or think about or impact my twenty-something day-to-day.

I also recently became a father. Becoming a parent is a singularity-like event. Those not there have no idea what it’s like. Those past the point of no return can’t get the message back out. It instantly changes the world you live in and the person you thought you were and what’s important and precious.

That’s how the issue of “work-life balance” went from being funny stuff old people talked about to front and center in my life. (I put the term in quotes because I hate it. But it has entered the parlance, so I’ll stick with it.) Here I just want to commit some thoughts to blog (in other words, a rant). I won’t pretend to offer any solutions, just observations.

First things first: this is not an issue for single people, and even couples without kids. Sorry. Let’s not even dilute the debate by including them in it. For them, achieving work-life balance is a nice-to-have, like that fancy car or exotic vacation. Sure things would be better with it, but it’s not exactly crucial. If the twenty-something me waxed eloquent about work-life balance to the present me, I’d smack him on the head. Your own self, or even you plus a partner, have infinite control over the contours of your time. An adult partner has way more flexibility in the timing and quantity of care and feeding, emotional or physical, they need from you. A baby, on the other hand, needs what it needs when it needs it, full stop. Needing to cancel Friday night club hopping because you had to crunch on your project is not in the same league as missing your toddler’s doctor’s appointment.

Ann-Marie Slaughter has pretty much written the last word on the topic in her Atlantic piece, Why Women Still Can’t Have It All. If you care about this issue at all, you have to read that it its entirety, and then let it sit in your head for a few days. It is insightful, thoughtful, and comprehensive.

But the debate is still largely gendered. Maternity leaves are much longer than paternity leaves. Mothers stressing out about their babies is considered normal, dads doing the same is considered saintly. Why the low standard for fathers? Only lately have I started reading about working dads and how this issue looks through the lens of fatherhood. The signs are encouraging. I see more dads around me “flaunt” their fatherhood more openly, and be less shy about openly declaring that they’ll take care of a certain piece of work after they’ve attended their kid’s school play.

I see corporate policies and environments, at least in the small tech echo chamber I inhabit, moving in the right direction. Managers are supportive of parents and their need for flexible schedules. Usually it is the parents who are hard on themselves, and can never outrun the feeling that they’re shortchanging both work and family.

There is an “invisible hand” in the modern workplace that values velocity above all else (and I can understand why), and creates pressure for more, more, more without any one individual or manager breathing down your neck. The paradox is that every single individual one interacts with at work might be reasonable in their expectations, yet, in aggregate, expectations always build up to where they’re overwhelming. The problem with velocity is that, when cognitive bandwidth is constrained (which is a fancy way of saying “when your brain is fried”) activity and just dumb number of hours put in is mistaken as a substitute.

To break out of this, one has to judge the value of work by the result and impact achieved, not the hours put in, or where they were put in. And that is easier said than done. A big part of modern work is defining what the work is. Time is thought of as a combustible resource–put more in, get more out–even if that has now shown to be not true for knowledge workers. Here’s a thought experiment: if you or your team achieved all your goals for the quarter in the first two months, could you all take the remaining month off? You’re probably laughing at me because the answer is “of course not, we’d just find more work to do in that extra month.” But 10x developers don’t pull off their magic by working long hours. They do it by figuring out what’s important and focusing on that.

I thoroughly enjoy my work. I don’t even usually think of it as work, and I know I’m fortunate in that respect. Stepping away from it is a struggle. But when I look back over the last few years, the times that stand out the most are my travels with my family. Everything else, work or not, pales in comparison. If I looked back again in a few years, what would I regret more: not putting in enough hours for that project (and maybe pulling down some more money), or missing my son’s first tooth or first steps or first words?

We report on an empirical study where we cut off email usage for five workdays for 13 information workers in an organization. …Our results show that without email, people multitasked less and had a longer task focus, as measured by a lower frequency of shifting between windows and a longer duration of time spent working in each computer window. Further, we directly measured stress using wearable heart rate monitors and found that stress, as measured by heart rate variability, was lower without email. Interview data were consistent with our quantitative measures, as participants reported being able to focus more on their tasks.

The results are not surprising, but it’s good to have solid evidence to back what many suspected. What is astounding is that the experiment got carried out at all.

Full paper here..

(The quote in the title is by Sherry Turkle, taken from the paper.)

Marius Eriksen tweeted me the slides for a talk by Tony Hoare that talks about how concurrent programs see reduced wait times, or “wait faster.”

The money slide:

(At this point you should be wondering what the probability distribution for arrival time is. That’s explained in the very next slide.)

And part of what makes them great are the rangers. Their love for the parks shines through, and that is what makes them want to make every visitor love the parks just as much as they do. I still remember the joy bubbling in a ranger almost as much as the geysers in Yellowstone as he gave us a tour.

So reading this account (in two parts) of how one ranger got the job was an interesting inside glimpse.

… “ranger” is lots of people’s dream job. (See last year’s Outside magazine jobs issue—this is referenced no fewer than three times.) Second, as far as I can tell, there is no “standard” way to come into rangering. This may be changing as there are starting to be university degree programs working in conjunction with NPS to funnel people into the profession via a specific college track, but right now literally everyone I know got here a different way.

Imagine that in the course of your day-to-day coding, you run into a problem that a teammate could help you out with. You revolve your chair around with a flick of your toes, look across your cubicle farm towards them. Most folks have their giant ear-muffler headphones on, the universal sign for “do not disturb, I’m trying to get stuff done.” But it’s your lucky day! The person you want to talk to is not wearing headphones! You walk over and present them with your difficulty.

At this point one of two things can happen. Either they know exactly how to solve your problem, right off the top of their head, or they don’t know or can only give you vague pointers. The former is much less likely than the latter. In the first case you go back to work, zipping along your task, the obstacle a distant memory in your rear view mirror. Your teammate decides, what the hell, might as well look at email now, and maybe gets back to work an hour later.

Zap to an alternative universe.

You decide to actually think about it, and dig around yourself a little bit before turning to your teammate for help. You know that if you’re truly up against a wall someone from your team will be able to help you. So you dig further into the code and into the docs. Maybe you learn about parts of the codebase you didn’t know much about. Maybe you find code smells and make notes to come back and clean it up. Maybe you write some docs about how to solve the problem, so that the next person has an easier time. And in the very worst case, if you absolutely must ask, your question went from “How do I do X?” to “I want to do X. I dug around and tried A, B, and C, but none of those seem to work. What am I missing?” And the latter usually elicits much kinder responses than the former.

Why do I spill so much virtual ink? The first scenario is typically what people have in mind when they talk about the advantages of a team working in close physical proximity. It’s a mode of working where synchronicity is the default. The thing is: it’s a myth. Usually when you put 10 programmers together in a room you get 10 people with giant headphones communicating with each other largely through instant messaging and email.

There is a limited and rarely occurring set of circumstances when strong synchronicity is required and productive. The best example I can think of is responding to a live production emergency that spans multiple systems and requires the expertise and input of multiple people. Another is in the early stages of design, when things are ambiguous. That style should be recognized as a powerful weapon to be used only when absolutely required.

The West has always had a fascination with Eastern philosophy. Remember the whole psychedelic phase of the Beatles? It has seemed like an exotic gateway into a realm not usually part of Western discourse and life. And especially once people’s material needs were well met in developed countries, they naturally turned towards more spiritual questions.

Today, you see it most in what I like to call “yoga culture”. Every yoga instructor has their own hour-long unique pill of exercise, poses and pop spiritual advice. As an Indian the very first thing that strikes me as amusing about it is the rampant mispronunciation of Sanskrit and Hindi words and pose-names. It reaches a peak when someone decides to recite an entire shlok in an Anglicized accent. All this is neatly ensconced a high-end retail environment. There is the constant undertone that buying these $80 yoga pants will make you a better yogi and nudge you along the path to self actualization. This strategy has worked wonders for Lululemon Athletica, rocketing them to the nearly the top of retailers ranked by sales per square foot per quarter.

It’s the same point the article makes: that being more mindful is a means to becoming a more productive worker, not an end unto itself. Somehow it’s hard to shake the feeling that all this eastern philosophy is an elaborate ruse to just sell us more stuff, spiritual or physical.

But there is a genuine need though. People feel stressed and anxious and beleaguered, and want to find a way through it. Like the doctor said to Don Draper, “people will do anything to alleviate their anxiety.”

To me the reason it all feels fake is that it’s all happy and pleasant. In Indian mythology, the idea of awakening or enlightened is inextricably tied to penance and suffering. Getting there is painful and hard work. The journey of every mythical hero has a phase where he or she realizes that to break through the barrier before them they need to go through a period of deep introspection and pain. The Hindi word is tapasya. (The two most common places for doing this are the woods, or for the really next level, the Himalayas. The Himalayas are to every other mountain range what a Ferrari is to a Ford Focus. You really have to see them.)

When you strip all that hard work and pain away, it loses all it’s potence. It becomes just another feel-good drug, putting you on just another cycle of suffering.

The design of CPU pipelines can teach us some deep things about how work should be structured.

— Vivek Haldar (@vivekhaldar) May 13, 2013

I think the modern CPU is an excellent parable for the nature of modern work itself. In a sense, it is the purest expression of Taylorism.

(Prerequisite: none of this will make sense without a basic knowledge of computer architecture and how CPUs are structured at a high level as a series of stages (called pipelines). The canonical reference is Computer Architecture, by Hennessy and Patterson.)

1. The goals of structuring the work are efficiency (utilizing resources optimally), latency (reduce the time it takes from start to finish) and throughput (maximize the work completed per unit time).

2. In order to achieve these goals, work is structured into a sequence of stages, with each stage performing a narrow, well-defined sub-task that is relatively easy and simple.

3. Work should not pile up. Inventory is anathema. There is a well-defined and bounded buffer between stages to hold intermediate results.

4. The latency and throughput is limited by the slowest stage in the pipeline. Thus, it is important that each stage be roughly equal in complexity and complete work in about the same time. This also helps to minimize temporary inventory (3).

5. Specialization is key. As a corollary of all the above points, each stage is naturally pushed to a narrow specialization.

6. Complexity in the overall system is acceptable, and indeed often inevitable because the above rules gravitate naturally towards a deep pipeline with a large number of stages, as long as each stage is simple.

The “golden copy” of my data used to be on local disk, remote copies being backups. Now things are a little different. In a cloud world, the golden copy is in the cloud. (In my specific case, “disk” is now Google Drive.) Files are not saved to local disk, they are saved directly to the cloud.

How should backup strategies and techniques change given the transition to the cloud?

The answer has to be motivated by the question: why do you need the backup? That sounds like a silly question, because, well, obviously you need the backup to avoid data loss in the event of losing one copy. But the deeper question is about the risk profile of your data’s copies.

In the large-local-disk scenario, the remote backup was protection against the array of physical risks associated with local storage–disk failure, accidental erasure, loss of laptop. In the cloud world, the risk profile is very different. If you’re using any decent provider, the chances of data loss in cloud storage are negligible. For one, cloud storage options usually employ some degree of redundancy. And more importantly, there is an an army of ops folks carrying pagers tending to the care and feeding of the machines and systems that contain your precious data. All this means that the chance of data loss is much, much smaller than local disk. Actual data integrity has taken a giant leap with cloud storage.

So what is the risk? It shifts, from loss of data to loss of access to data. They sound like the same thing, but the difference is that data loss is permanent, but loss of access may be temporary. There might be service outages. Your wifi or ISP might go down. You might forget your password. The cloud company might go out of business and you might not be able to get your data out in time.

How do you mitigate against these risks of cloud storage? Periodically make a full local copy! We’ve made a full reversal. From local disk as golden, to local disk as backup. This is progress overall, because I think the probability of data loss in a cloud world is much, much smaller, and also the entire setup is much simpler. There is also the incremental but very nice benefit of much higher chances of survival of non-backed-up data (that is saved to the golden source but not to backup yet), because the probability of your single disk failing is much higher than that of a cloud service losing your data.

The idea of hypertext was also an active research area at the time. Look at the proceedings for the ACM Hypertext Conference 1989, and you will find a plethora of papers delving into what the structure of hypertext documents should be, how they should be used, how linking should work etc. Going forward and backward through other years of the same conference will show you more and more variations and details on all aspects of hypermedia.

HTTP and HTML bucked several established ideas that the academic community was convinced were good ones, if only they could find neat implementations of them. His crude implementation must have looked primitive for several reasons. For example, links were not brokered by some central entity, and there was no guarantee of them actually working. They could be incorrect, or simply break if the target moved. Quotations did not automatically preserve links back to the source. And so on. Of course, it was these very characteristics that enabled it to scale and spread without any central points of failure or bottlenecks.

The doomed Project Xanadu tried to bring several of these ideas to life, but failed, arguably under the weight of its own complexity. Consider the idea of canonicalizing the act of copying, and what a strain it was both conceptually and in terms of implementation and resource consumption:

Xanadu was to consist of easy-to-edit documents. Links would be available both to and from any part of any document. Anybody could create a link, even in a document they did not write. And parts of documents could be quoted in other documents without copying. The idea of quoting without copying was called transclusion, and it was the heart of Xanadu’s most innovative commercial feature - a royalty and copyright scheme. Whenever an author wished to quote, he or she would use transclusion to “virtually include” the passage in his or her own document.

Nelson was frequently frustrated by his failure to convince casual questioners of the importance of his transclusion idea. Transclusion functions like the “make alias” command familiar to Macintosh users. An alias works as a fully functioning copy of a file or application, but it is really just a pointer, or virtual copy. Click on the virtual copy, and the original file or application begins to run.

The key to the Xanadu copyright and royalty scheme was that literal copying was forbidden in the Xanadu system. When a user wanted to quote a portion of document, that portion was transcluded. With fee for every reading.

Transclusion was extremely challenging to the programmers, for it meant that there could be no redundancy in the grand Xanadu library. Every text could exist only as an original. Every user in the world would have to have instant access to the same underlying collection of documents.

Of all the design axes of a hypermedia system, only two ended up mattering: simplicity, and scalability.

Of course, hindsight is perfect. But it takes courage and vision to neglect all the bustling activity in a field and pull out a tiny sliver of it–the right sliver.

Betsy DiSalvo (now an assistant professor at Georgia Tech) starts from an interesting observation. Many computer scientists (who are mostly white or Asian, and male) say they became interested in computing because of video games. No demographic group plays more video games than African-American and Hispanic teenagers and men. But few African-American and Hispanic males become computer scientists. Why was that?

DiSalvo explored her question with ethnographic methods. She observed African-American teen males playing video games and talked to them about how and why they played. She found they were playing video games differently than white teen males. Her participants never used “cheat codes” or modified their games in any way—they used video games like athletic competition. Manipulating the football or the field is cheating, so why would you change the video game?

The full paper is here.

Here are some (paraphrased) points from his dissertation that I found particularly interesting:

• There were three main conceptions of CS among students (with the percentage that held it):
  1. Programming is central, with other subfields in supporting roles (41%)
  2. Broad view: CS consists of many computing-related subfields (27%)
  3. Theory: CS as the mathematical study of algorithms (8%)
• As ever, there is still a deep tension between the programming and implementation-centric and algorithms and theory-centric view of CS. Theory still has a PR problem. “Many students did not understand the purpose of CS theory, and a few left it out of their description of CS entirely.”
• At the high-school level, application use was conflated with CS. E.g. believing that a CS degree could be used for a Photoshop job.
• Surprisingly, misconceptions about CS did not change long-term educational decisions, which were based on more immediate experiences like whether a course was enjoyable.

This is a battle that had been fought since the 60s, and for all practical purposes that chapter is closed now.

So I wonder, what will programming languages look like in 2050? What are the big problems we should be tackling?

I will put on my prognostication (or make-a-wish) cap, and if that makes me look like a fool down the road, so be it.

Better Systems Programming: functional constructs made it into imperative languages, but lower-level systems programming is still stuck in the 70s. In good ol’ C. Languages like Go are trying to change that. But systems programming (by that I mean stuff like operating systems and virtual machines) is still done by a priesthood with an extremely high barrier to entry and experimentation. The last and only project I ever came across that tries to lower the barrier for experimenting with operating systems was OSKIT from the University of Utah. Imagine the flowers that would bloom if programmers could try out OS ideas with the same investment of effort as writing a Python script.

The current landscape has me pessimistic, for the same reasons Rob Pike laid out in Systems Software Research is Irrelevant: that to even get to the starting line you have to support a huge mass of legacy stuff for compatibility.

One project I like is Microsoft’s Singularity, an attempt at building an OS from the ground up in a safe language.

Treating large bodies of code as raw data for machine learning: Imagine a Clippy for programming. What if your IDE went: “Looks like you’re trying to write a for loop over this array, but you have an off-by-one error. Would you like me to fix it?” Or: “Looking at the signature of this method, here are the most likely methods you will need next to get to the type of the returned value.” That’s your editor understanding the semantics of your language and code base at a deep level.

The fun starts when you combine that with the millions of lines of openly available code and bug reports, feed it to a giant machine learning system, and build up machine knowledge about bug patterns, patterns of good and bad code, idioms and best practices (“Looks like you are trying to filter values from this list. A better way to do that is…”).

Think of it as coding with an entire datacenter behind you to help.

Hardware will eat software from the bottom: Has been a long trend, and I don’t see it stopping. Common low-level software patterns gradually make it into the chip. Network and graphics cards have long been absorbing higher and higher level functionality into silicon. The latest example that’s quite exciting is Intel’s version of transactional memory in hardware, hardware lock elision.

Laptops and desktops–I put them in the same category because they are heavy, power hungry, and have full size keyboards–are factory machines. They are what I use to get serious work done, the kind that gives me a paycheck. I almost never reach for them outside work situations.

On the other end of the spectrum are phones. I don’t know why we still call them phones because I never use them to make phone calls, because in general I never make phone calls. They are rude, disruptive, and synchronous. There is a single digit number of people on the planet from whom I will accept phone calls. But these “phones” are perfect for being our assistants out in the physical world, where we need maps and directions and traffic updates. And that’s mostly what I use my phone for. I almost never use it indoors. Except as a camera. Because the crappy camera you have at arms reach–and they’re getting less crappy by the day–beats the expensive one in your closet.

Which leaves the great big middle of… everything else. This is where the tablet comes in. It has become my main non-work device. I use it for reading–tons of reading. I use it for writing. Gesture typing on Android has greatly improved the speed and accuracy with which I can heave words into the device. I probably spend as much time on the N7 as my laptop/desktop combined. And I think it will only grow. If we start seeing new interesting interfaces that make it viable to actually code on tablets–and we’re beginning to–that will be another blow to laptops and desktops.

It’s a floppy disk. There is only a tenuous connection between saving and a floppy disk even for those of us who know what a floppy is (and at the moment most of us remember them), but floppy disks are on their way to becoming as unknown as Charles Yerkes. Don’t know who I’m talking about? That’s my point.

Floppy disks were a stepping-stone medium — once ubiquitous, they have given way to larger, faster, and more convenient forms of storage. Soon, nobody is going to remember floppies, except for those of us re-living the good old days when we used to replace their magnetic sheets with sandpaper as a practical joke. When the new generation of users takes over, they’ll have no idea of what a floppy is, and the icon will have lost all meaning.

His larger point in the post is about using words rather than symbols for icons, but I want to come back specifically to the idea of “saving” documents.

The idea of creating a digital artifact on a computer and it being ephemeral until one remembers to “save” it to a more persistent medium has caused countless people untold misery since the dawn of the computer age. I haven’t met anyone who does not have their own personal story of loss. Inspiration strikes, you heave a thousand words into the computer, and in the heat and fluidity of the moment forget to save your precious words. And then, the computer crashes. Noooooooo….

But, progress happens.

In most web apps, the concept of saving has simply gone away. You type your stuff, and the cloud captures it for posterity. You do not have to remember to save anything. At least, Gmail and Google docs work this way. This, of course, is how it should be.

I’d go as far as to say that if an app made today has the concept of saving work, it is fundamentally broken and does not belong in the 21st century.

Well, I loved this movie. I loved the way Coppola and her actors negotiated the hazards of romance and comedy, taking what little they needed and depending for the rest on the truth of the characters. I loved the way Bob and Charlotte didn’t solve their problems, but felt a little better anyway. I loved the moment near the end when Bob runs after Charlotte and says something in her ear, and we’re not allowed to hear it.

We shouldn’t be allowed to hear it. It’s between them, and by this point in the movie, they’ve become real enough to deserve their privacy.

That’s Roger Ebert on “Lost in Translation”, a movie that had me wandering in my head for weeks afterwards.

Often we are at a loss to understand and express why something touched us. We feel the sharp jab in our souls, but lack the words. “I loved that movie!”, is all we can say, flaccidly.

Roger Ebert filled that void. He gave words to feelings many of us didn’t know we had, that were stirred by movies. We often mistake a movie review as pure opinion, as something everyone is entitled to his own distinct view of. But Ebert touched on the underlying universal constant that is the same in all of us.

That was his power as a human who felt, and then magnified that using his words.

So here they are, all the long-form posts from this blog, categorized as best as I can. Start here.

Mad Men is an honest portrayal of the themes of modern work in all its glory and pain.

I once described the show to someone who hadn’t seen it as a long, detailed and deliberate character study of one person: Don Draper. Now I realize it is more than that.

The lens through which that character study is done is his work. The main stage of the show is the office, with home filling in the interstices. The primary way in which characters relate to each other is through their work.

Along the way, Mad Men starkly reveals truth after truth about the nature of modern work. They are part of our daily life, but still so hard to see. We’re blind to them, just like a fish would be at a loss if asked about the nature of water.

From building the physical to creating the imaginary

Look at your hands – they’re as soft as a woman’s…What do you do? What do you make? You grow bullshit. – Don’s father, to Don.

Modern work deals largely with imaginary, sometimes ephemeral artifacts. Even where the final output is a physical object (a chip, a phone, or a car), the physical object is but the last link in a long chain of platonic objects. Often, that last link is not even thought of as “work”, but as “assembly”, ideally handled by robots.

3D printers now make it so that we can think of that last step as “compiling”. Just like you compile source code to get an executable, you compile your 3D design to manifest it into the world of atoms. The compiler is just another piece of automation.

Roy: Perpetuating the lie. How do you sleep at night?
Don Draper: On a bed made of money.

We use our hands not to move and arrange physical objects, but to command the manipulation of bits. At the beginning of this transformation, it was understandable for the real makers, the movers and arrangers of physical things, to look down upon people pushing around paper in an office and wonder, just like Don’s father did, ”What do you do? What do you make?” Don is condescended to even by the “true creatives”, the starving bourgeois artists.

Some might still wonder, but the derision is gone (or at least, veiled), after the complete shift of power and wealth and attention from the old physical makers to the new bit-wranglers.

The War for Talent

Don Draper: It’s your job. I give you money, you give me ideas.
Peggy Olson: And you never say thank you.
Don: That’s what the money is for!

Each factory worker was more or less like every other factory worker. But a star creative director or copywriter or programmer is irreplaceable, and hence, valuable. If you lose them, you might, after much effort, get another great one to fill their position, but they will not be the same. They will have a different creative signature. Like a uniquely shaped diamond, each demands their own custom-made setting.

Don picked Peggy from the ranks of the secretaries and made her a copywriter. He challenged her, and forced her to grow. He was mean to her. He was tender and protective towards her. In the entire cast of characters, she is the one who understands (or comes closest to) what makes Don Draper tick. But while Don was a great grower of talent, he was not a great retainer of talent. Don eventually loses Peggy to a rival ad agency.

The overt reason is that she got a much better financial offer. But that is not the true reason. It’s not (only) about the money. It’s about what the money signals. Peggy doesn’t live a lavish life. She’s only in her apartment to sleep. She doesn’t have plans to spend all that money. But she does want to be perceived as valuable. She wants to be wooed. And Don did not give her that. The head of the other agency, a long-time rival of Don, spent time and effort trying to win her over. It was that treatment she fell for.

The Primacy of Work

I know what I’m supposed to want, but it just never feels right, or as important as anything in that office. – Peggy Olson

All the characters in the show derive their self-worth through their work. All of them. There is not a single character that has any significant source of meaning outside their work. Those that have families are cast as providers and protectors rather than lovers and nurturers.

Peggy goes through a string of half-hearted relationships, each of them sacrificed to work. But the sacrifice is a willing one, not forced. There is a brief twinge of… what? Remorse? Regret?… before diving eagerly head-first back into work, into the comfortable rhythm and embrace of it.

Work is exciting. The office is a dynamic place. It is a place where you can reinvent yourself. New projects. New titles. New responsibilities. It is a place that always looks towards the future. It is sterile enough to be a place where you can regularly be born again. Home, on the other hand, is an anchor, a place where you are tied to your past, where you can never run away from the old you.

This is why Don’s relationship with Megan is an outlier, the curveball of the entire show. He does truly love her, more than any other woman in his life. More importantly, he respects her. While he didn’t think twice before cheating on his first wife, he actively shuns overtures from other women to remain faithful to Megan. Megan might be approaching, or even surpassing, the meaning and fulfilment he gets from work. And he has no compass to deal with that. It is uncharted territory for him. Which is why flashpoints with Megan leave Don (and the viewer) confused, but always end up reinforcing how much she means to him.

But at the end of it all, it is not enough. Whatever satisfaction and sense of worth work gives them leaves them searching for something else. They are anchorless, tumbling through dark waters, guided by faint and distant lighthouses. Whether they see a light or not, we will eagerly follow, if only to get clues for ourselves.

• explain why it’s needed
• minimize the resources it locks
• hold it for as short a time as possible

And then go about your work.

1. What is really hot in STEM jobs?
2. The Catch-22 STEM job market
3. What do STEM employers want?

The key results of this brief survey of the Craig’s List San Francisco Bay Area job board are: very few entry-level or junior STEM jobs are posted. Most STEM job posts claim to require 2-7 years, often 3-5 years, of work experience in specific skills such as programming languages such as MATLAB or C++ and toolkits such as OpenCV or OpenGL. Very few jobs requesting more than 10 years of experience are posted. There is little or no interest indicated in a range of mathematical skills taught in high school and college math including algebra and especially calculus.

STEM job posts with a high mathematical content are dominated by statistics and data analysis, primarily business data and some medical/healthcare data. Most machine learning, “big data”, and data scientist posts fall into the category of statistics and data analysis. There are remarkably few STEM jobs posts seeking to solve “big problems” such as alternative, cheaper energy sources, curing major diseases such as cancer, and the like. The few companies that arguably post “big problem” jobs are often backed by the government (e.g. Elon Musk’s Tesla Motors and SpaceX) or may have special relationships with the government (e.g. Robert Bigelow’s Bigelow Aerospace).

Many of these results contradict common claims and themes in general news media articles, think tank reports, and other sources about STEM education, STEM employment, and alleged shortages of STEM workers (scientists, technologists, engineers, and mathematicians).

The counterpoint is made by this Senate Committee Report:

…the demand for STEM-skilled workers is expected to continue to increase in the future, as both the number and proportion of STEM jobs are projected to grow. New Bureau of Labor Statistics data show that employment in STEM occupations is expected to expand faster than employment in non-STEM occupations from 2010 to 2020 (by 17 versus 14 percent). In addition to government projections of employment growth in STEM fields, business organizations and other groups have issued numerous reports and surveys that suggest that there is a heightening need for qualified STEM workers – both those with highly specialized skills as well as those with a more general knowledge of STEM concepts. For example, even at the height of the recession, a survey of manufacturers found that over one-third were experiencing shortages of engineers and scientists – and most of them anticipated greater shortages in the future. In addition, an earlier survey found that over half of manufacturers believe that the public education system insufficiently prepares students with the math and science skills necessary to succeed in the workplace. Furthermore, concerns regarding shortages of skilled workers are compounded by the pending retirements of many baby boomers, an issue cited by both private industry and government officials.

So which is it? A bit of both, I think.

Job listings are largely fantasies. Just because someone wants a domain expert with 10 years of experience doesn’t mean they’ll get one. Often, employers truly care about a small but important subset of the long list of “requirements” in a job listing.

There being little demand for workers to tackle “big problems” does not imply that there isn’t a shortage of STEM workers. That might simply be a reflection of the fact that it takes huge teams to tackle large problems, and there are rarely listings for “come cure cancer” or “come build the next-gen vehicle.” Open positions will likely focus on small niche sub-problems, which don’t seem as big and challenging.

I encountered Goldwasser’s work during a graduate course in cryptography. I came across the idea of zero knowledge proofs, which is a technique for Alice to prove possession of a secret to Bob without actually giving away the secret. At the end of their interaction, Bob will be convinced that Alice possesses the secret, but will have no idea what it actually is. At first, this sounds completely paradoxical. How is that even possible? But it is.

Things like this still make me feel like crypto is the quantum mechanics of computer science, full of counter-intuitive and mind-bending results.

Here’s a talk I gave to the class back then. Deepest apologies for the chintzy PowerPoint theme. I blame it on youth.

I didn’t start out with the goal of devoting all of myself to my job. It crept in over time. Each year that went by, slight modifications became the new normal. First I spent a half-hour on Sunday organizing my e-mail, to-do list and calendar to make Monday morning easier. Then I was working a few hours on Sunday, then all day. My boundaries slipped away until work was all that was left.

Workaholism is the perfect term for this, because, like alcoholism, it connotes addiction. Nobody starts out wanting to become an alcoholic, or a workaholic. But you start down a path, and one fine day you wake up with a dry mouth and a splitting headache with no memory of the night before, or realize that you haven’t seen family or friends in days or weeks. You think you are in control of it, but eventually it is in control of you.

It happens through a series of small choices, each one seemingly inconsequential. And it is a choice, because I’ve rarely seen a workaholic creative professional doing 80-hour weeks because their boss was tightening the screws on them. They did it to themselves.

But why did they choose this? Not for the money. Workaholics are usually already well-paid, and hardly ever enjoy that money anyway.

What is the hook? Where is the high? It is the need for cognition, a trait that compels one to “understand and make reasonable the experiential world”. Every problem solved, every little checkbox marked off, every little question answered, every gap covered–these are all highs. And when the pile of work is infinite, and usually self-created, you can string together a series of such highs into a sixteen hour workday.

But just like addiction, workaholism is an easy way out. An easy way to ignore the work and time and compromise required for cultivating relationships. An easy way to pretend that there’s simply no time to step back and distinguish the urgent from the important. An easy way to distrust colleagues and do it all yourself. An easy way to deal with the stream of work as it arises, flowing in it, rather than putting in the time and deep thought required to prioritize and tackle items with potentially high impact.

Now Cade Metz over at Wired has saved me (and a bunch of others, I’m sure) by managing to go a bit deeper into what that means, adding some color to The System That Shall Not Be Named.

This software system is called Borg, and it’s one of the best-kept secrets of Google’s rapid evolution into the most dominant force on the web. Wilkes won’t even call it Borg. “I prefer to call it the system that will not be named,” he says. But he will tell us that Google has been using the system for a good nine or 10 years and that he and his team are now building a new version of the tool, codenamed Omega.

Borg is a way of efficiently parceling work across Google’s vast fleet of computer servers, and according to Wilkes, the system is so effective, it has probably saved Google the cost of building an extra data center.

What pushed me over the edge was the Pixel.

I will not review the Pixel here. Plenty of folks have done that in depth already. I will just say that the hardware is delicious. Particularly the screen. I’m a bit of a typography nut, and seeing type on a screen of that resolution makes you realize what a waste it was looking at and picking out fonts for low-resolution screens. I now hate to connect to a large external monitor because it just looks so crappy.

(By the way, I’m biased. See my “About” page. Take all this for what it’s worth.)

Here I want to talk about how some of my workflows and habits have changed (or will need to change). I’ve had the Pixel as my full-time sole machine for a glorious four days now, so a lot of this might change as I find my way.

Whether ChromeOS is right for you depends entirely on whether you live on the web full time, or are still shackled to native apps. Even if you do live on the web full time or most of the time, like I do, when you are on a “regular” machine you fall into habits that reinforce the use of the local machine as a crutch.

There are two major factors that might tie someone to a “regular” machine. The first is native apps. In the spirit of aiming for where the puck is headed, I feel confident that those who are absolutely tied to native apps will continue to dwindle and be confined to highly specialized niches. High-end image editing, graphics, 3D rendering and video editing is the most obvious one.

The other major reason is the use of local disk. That gargantuan terabyte (or fraction thereof) of space attached to your machine–but only your machine. Local disk ties you to specific machines. Local disk makes you worry about backups. Local disk fills up. Local disk is 20th century. On my previous main machine (a 13-inch MacBook Air), I used local disk like a messy floor. The things I cared about would always find a place online, usually on Google Drive or PicasaWeb, and those that I was unsure about or were temporary in nature would get tossed on the floor, usually never to be looked at after the first time.

Even though the Pixel hardware is not cheap, there is a certain peace of mind that comes from knowing that a ChromeBook is disposable in the sense that I don’t have to worry about “all that stuff that was on that machine.”

That said, I plead guilty to being addicted to local disk. A lot of crucial things in my setup depend on local disk. Thankfully, they don’t have to. It’s a legacy workflow that will take me awhile to properly untangle. The first step is to use “Save to Google Drive” when “downloading” something from the web.

So how have my workflows changed? The big news is that they haven’t, for the most part. I used to spent 90% of my time in the browser before, and now I spend 100%.

The biggest thing I thought I would miss was dictation software. I used Dragon Dictate on the Mac. But the spanking new WebSpeech API looks like it will at least start to cover some of that ground. Check out this demo of speech to text in the browser.

There are still some odds and ends I miss. I got used to a bunch of trackpad gestures in MacOS X that don’t work on ChromeOS. Two and three-finger swipes, mostly. I’d also developed a deep habit of using sticky modifier keys (that’s when you can press and release “Ctrl” and then press “A” to emit the “Ctrl-A” key combo, rather than having to hold down “Ctrl” and then pressing “A” while “Ctrl” is held down). I still miss that.

But so far, living full time in the cloud has been great. It’s the future. Might as well start now.

“Son, in my time we wore eye glasses without computers in them.” “Dad, you talk funny.” #

“Son, in my time we had to go to another place and sit there for 8 hrs everyday to earn money.” “Dad, you talk funny.” #

“Son, in my time we learned logarithms and looked up logarithm tables to do complex calculations.” “Dad, you talk funny.” #

“Son, in my time we traveled long distances to touch other people.” “Dad, you talk funny.” #

“Son, in my time I could multiply two three-digit numbers in my head.” “Dad, you talk funny.” #

“Son, in my time we used tiny sticks that squirted ink to make letter-like patterns by hand.” “Dad, you talk funny.” #

“Son, in my time we had to stay awake and drive these cars ourselves.” “Dad, you talk funny.” #

“Son, in my time we used to punch in words one letter at a time on boards with keys.” “Dad, you talk funny.” #

I don’t see them completely going away. Very high-end work (video, graphics, big compiles) will probably always want something that’s hotter and faster than what can fit into a thin slab.

To someone who didn’t grow up with them, desktops will look like factory machines. Heavy, ugly things that don’t belong in the everyday personal world, but solely in places of work, serving highly specialized requirements.

Keyboards and trackpads are headed the same way, I think.

Update: Right after hitting “Publish” on this post, I saw this post by Chris Dixon (off HN) which makes more or less the same point.

What would you think of such a person? What kind of a statement are they making?

It is a renunciation of the digital in favor of the analog. They’re giving up on making their notes searchable and linkable and shareable. This is the sort of voluntary shift away from digital and towards the material analog that Bruce Sterling has characterized as the mark of the rich in today’s world. Or conversely, too much connection is a sign of poverty.

They’re saying that they’re not writing it down to remember it later, they’re writing it down to remember it now. Maybe they’re important enough or smart enough to not have to grub through their notes from the last meeting (like you were doing on your laptop, trying to get some context).

It is a status symbol. Just like watches became jewelry in the 21st century. Because why even go to the trouble of having a fountain pen if it isn’t an exquisite, expensive one?

It is a sensuous oasis in a homogenous environment. If you’ve used one, do you remember the sensuous joy of writing with a fountain pen? The smooth slide of the nib on the paper, tempered just a little by the friction against it. The trail of wet ink, gleaming, drying a few seconds later. You could even smell it. You didn’t want to run out of words, because then you’d have to stop writing. It was a calming, almost meditative experience.

The same line of thought holds true for many analog things. Things for which we must leave behind the ethereal digital and embrace durable atoms. Typewriters that are keyboards. Face-to-face gatherings of humans. Large, heavy wooden desks. A thick hardbound book, with marginalia and bookmarks and its very own smell.

What is the mark of luxury? Rarity. Price. Being understated and not ostentatious while still making a bold statement and attracting attention. The analog physical will possess all of them. As we dive further into the digital, and as most of it becomes commonplace and cheap, coming across something from the old analog world will seem all the more striking.

Analog objects will be the new luxury.

Productivity has been on my mind (again) lately. I present some thoughts here, if only to get a load off my head.

Productivity is for peasants. Yes, that is a provocative framing, but the point stands: productivity is you being accountable to someone, and them measuring you by metrics they deem appropriate. The aristocracy in ages past explicitly shunned productivity. Not doing anything and living a life of leisure was the true mark of high class.

Productivity measurement made sense for factories; it doesn’t for offices. The notion of “productivity”, and of measuring it, is a Taylorist invention. Break work down to its individual components, split it out into an assembly line, and then measure each section, and the whole. The idea is firmly rooted in the separation of thinking (design of the assembly line) and doing (operating machinery on the line). Creative and ambiguous endeavors defy this scheme.

Measuring productivity is impossible. This is a stronger statement than saying that it is just hard. No matter what metric you pick, it can be perverted, and often is. Bugs, lines of code, hours worked, even revenue. These are consequences of being productive, sometimes, but one shouldn’t confuse them for the real thing. The root cause of this is that measurement and the numbers that emanate from it inherently lead to short-term local optimizations. For example, look at the all the pointless day-to-day activity around the stock market. True impact comes from long-term thinking, and while data can certainly inform that, it takes deep maturity to insulate oneself from small movements.

Productivity is about finding bottlenecks. Almost every measurement of productivity, as I’ve noted above, is meaningless. They don’t tackle the bottleneck of a system. Trying to optimize the parts of a system that are not bottlenecks is a complete waste of effort. Whatever purported gains are reported are probably due to the Hawthorne effect. So, first, find the bottleneck (at the level of the individual, team, factory, company, or whatever), then optimize just that one thing, until something else becomes the bottleneck.

Productivity is not work done per unit time. My brain goes blank when people talk about “productivity”, in the same way a scientist might tune you out if you say something like “This stone weighs 20.” 20 what? Kilograms? Pounds? Everything I’ve read about productivity focuses on some sort of rate, a measure of work done per unit time. That is a local measure. It’s also the wrong unit. Productivity should be a dimensionless, unitless fraction between 0 and 1, representing how much of your potential maximum output you can achieve. (I suppose destructive people could have negative productivity.) Yes, I realize that makes it a relative measure that can’t be universally compared. Yes, like I said before, it is still hard to measure.

We should focus on work design, not productivity. Universe A: you work 40 hours a week, make 80k a year, spend evenings and weekends with family and friends. Universe B: you work 80 hours a week, make 200k a year, neglect your family and have no friends. You will live long and be happy in Universe A, and Universe B will make you miserable and prematurely kill you. From an individual standpoint, design of life and work yields much higher returns than maximizing productivity. (Remember that when you spend those hours on “tips and tricks.”)

The 21st century will be about defining “work”. The sweep of modern history so far has been about defining the common economic and political systems that civilization can live with. The various schools of thought duked it out over the last couple of centuries, and capitalism (tempered to various degrees by local taste) and democracy won, enshrining the idea of individual rights and liberty. It has now brought us to the point where the focus now turns inwards to what the individual should do. Work has now become tied to an individual’s self-worth. What does that mean if your job is automated away?

For example, consider this right-angled triangle…

… and two ways of stating facts about it.

The old way (quoting from the book):

1. The angle BAC, together with the angles BCA and ABC are equal to two right angles, or twice the angle ABC.
2. The angle CAB added to the angle ACB will be equal to the angle ABC.
3. The angle ABC is greater than either of the angles BAC or BCA.

And the new way:

I loved geometry in high school, but it was despite the drab textbooks. Why isn’t this the standard way geometry is taught everywhere?

You can download the entire book, or view it online.

(via Public Domain Review)

As the title suggests, a big part of the book is about the discontents of being enveloped in technology, and of spending your life building it. Like a fine Greek tragedy in which joy and sorrow are weaved fine, that is inextricably linked to what draws them in in the first place. When doing that it’s easy to fall into the trap being nostalgic, of looking back with rose-tinted glasses and discounting the advantages of the present, and favouring the past simply because it was known and loved. Deep in their heart of hearts, everyone’s ideal world is that of their childhood. That’s the trap of nostalgia– to think that the world was simpler and purer, when actually it was us who were simpler and purer. Which is why it was refreshing to see the author take an unsentimental look at the side of technology that estranges us, makes us “virtual.”

I thoroughly enjoyed it.

You could spend hours diving into the data and drawing your own conclusions, but these are some of the things that jumped out at me:

1. 7% of new PhDs got into tenure-track faculty positions.
2. 47% of new PhDs went to industry.
3. 57% of PhD enrollments were nonresident aliens (an all-time high), as were nearly 50% of PhD graduates.

What does this mean? Let’s take them in turn.

1. The tenure-track faculty position is the top of the academic totem pole and coveted by fresh PhDs. The hard truth is that those positions are small in number and getting smaller, leaving the very distant second choice of post-doc as the only viable option for those who want to stick with academia:

   The data indicates tenure-track faculty positions for new Ph.D.’s have declined steadily since 2004 from 224 to 124 in 2011 while the number of postdoc positions greatly increased. In 2011, new Ph.D. graduates accepted twice as many postdoc positions as tenure-track positions. Around 2003 there were approximately 2.5 times more tenure-track positions than postdoc positions.

   Which brings us to…

2. At least in computer science, the slack has been picked up in large part by industry. This might have been looked down on in the past (“I have a PhD! I can’t work as a money-grubbing corporate stooge!”), but as I’ve written before, the avante-garde tech companies of today offer work challenging and varied enough (and yes, well-paying enough) to keep nearly half of the graduating crop of PhDs happily employed. I expect this fraction to only grow in the future.

   1 and 2 combined should make every prospective PhD student do a good deal of soul-searching to figure out why exactly they want to go through with it. It is a big commitment, and it is not easy, not just in the intellectual and academic sense, but also psychologically. This doesn’t have to be done before you start. Remember that there is an “out” after about two years when you can leave with a master’s degree. I’ve reached the conclusion that there is no material advantage to doing a PhD, and that the only good reasons to do it are philosophical and deeply personal. I will always remember it as the last still, contemplative and reflective time in my life. A font that forever gives.

3. To anyone who has walked the halls of a CS department (especially at night), this statistic is far from a surprise. To most of the public, it probably is. Yes, the majority of people in CS PhD programs in the US are foreigners. And the vast majority of them get a full scholarship for the six to seven years that it takes to get a PhD. (They repay part of that by keeping the academic engine churning, but still.) The vast majority of that scholarship money comes from government grants (i.e. taxpayer dollars) from agencies like the National Science Foundation, Department of Homeland Security, Office of Naval Research etc. I am loathe to get into politically charged topics, but given this, it seems downright stupid to make legal residence in the US of those graduating PhDs so hard that a non-trivial fraction of them leave. The logical taxpayer should insist that they stay here and contribute back as productive members of society.

The abstract:

It is challenging for service providers to keep the tail of latency distribution short for interactive services as the size and complexity of the system scales up or as overall use increases. Temporary high-latency episodes (unimportant in moderate-size systems) may come to dominate overall service performance at large scale. Just as fault-tolerant computing aims to create a reliable whole out of less-reliable parts, large online services need to create a predictably responsive whole out of less-predictable parts; we refer to such systems as “latency tail-tolerant,” or simply “tail-tolerant.” Here, we outline some common causes for high-latency episodes in large online services and describe techniques that reduce their severity or mitigate their effect on whole-system performance.

Go read the whole thing.

Anil Dash:

High schools have long offered vocational education, preparing graduates for practical careers by making them proficient in valuable technical skill sets which they can put to use directly in the job market right after graduation. Vocational-technical schools (vo-tech) provide trained workers in important fields such as healthcare, construction trades, and core business functions like accounting. For a significant number of my high school peers, vo-tech was the best path to a professional job that would pay well over the duration of an entire career.

Now it’s time that vo-tech programs broadly add internet and web technologies to the mix. We need web dev vo-tech.

Laurie Voss:

Think about how a physical factory worked. The reason unskilled jobs in manufacturing, say, cars existed is because some very highly skilled people first got together and looked at the process of building a car and said “okay, we can automate this bit and this bit and this bit, but making machines to do this bit is way too hard”. The blue collar workers of Detroit didn’t know how to build cars: they knew how to operate one particular bit of machinery. The only people who knew how to build cars were the guys who put the machines together.

Now let’s try applying that model to web development…There is an entire industry built around cranking out simple websites for small businesses, in WordPress or Drupal or a thousand proprietary solutions… It’s as if Detroit were a city of craftspeople who built one car at a time from scratch in their garage at home – with love and care, but slowly and enormously expensively

Well, it certainly isn’t for lack of trying. You have to remember that since its very inception, the field of software engineering has lamented that it was not industrialized enough. At the 1968 NATO conference, Doug McIlroy said:

Software production today appears in the scale of industrialization somewhere below the more backward construction industries. I think its proper place is considerably higher, and would like to investigate the prospects for mass-production techniques in software. In the phrase “mass production techniques,” my emphasis is on “techniques” and not on mass production plain. Of course mass production, in the sense of limitless replication of a prototype, is trivial for software. But certain ideas from industrial technique I claim are relevant. The idea of subassemblies carries over directly and is well exploited. The idea of interchangeable parts corresponds roughly to our term `modularity,’ and is fitfully respected. The idea of machine tools has an analogue in assembly programs and compilers.

What the two gentlemen are talking about is bringing full-scale Taylorism to software development. Industry loves Taylorism! It would bring Taylorism to every possible activity it could. And yet, there is something about the activity of constructing software that has not yet relented to Taylorism. Some parts have, though. The “knowledge factories” that churn out websites by picking and choosing from a menu might not exist, but something similar has happened on a much larger scale with enterprise software. That is the business that every large IT services and consultancy company is in.

The problem with this vocational tech approach is that it implicitly assumes the knowledge gained during vocational training will be relevant for a long period of time. The pace of technology change renders that unviable.

Also, being a blue collar coder is not a sustainable career. The kinds of jobs envisioned for them are one skip away from becoming fully automated. Why even involve a human? Why not have an automated service where you can select items from a menu, and presto, you have a website for your small business? Being in that position means that your only bargaining chip is controlling aggregate supply of labor, i.e. unionizing, which will likely only add to the motivation for finding automated solutions.

Little known is the fact that programming started out as a field that was Taylorist to its core. The very first computer “programs” were specified at a high-level by scientists, and then the actual coding into computer instructions was left to… coders, literally. And at the time, that job fell on the secretarial staff, most of whom were women. The majority of the very first programmers were ladies. (Strange how the gender ratio flipped over time.) In time, the higher-ups realized just how much talent and brain-power it took to “code”, and how much was strangely left unspecified in the gap between the white-collar specifications of the managerial layer and the nuts-and-bolts “code”.

They tried making coding a blue-collar job from the very start, and it didn’t work[1].

I can’t help but think of it as a whale coming up for air, or a bear coming out of hibernation.

When someone asks you, “What’s the culture like?”, or “Is this candidate a good culture fit?”, what do they mean, and how do you answer them? Phrases like “work hard, play harder”, “move fast”, “risk-taking is rewarded”, “dynamic and collaborative” (who isn’t?) sound catchy and look good on recruitment pages, but are meaningless. What is it then? A list of rules? Some pre-requisite mental and attidudinal posture for blending in and functioning well in the company?

Culture cannot be spelt out. Almost all attempts to do so result in bland lists, which, if followed, would not resemble the real culture at all. Good cultures are self-propagating and strong, in that new hires quickly ingest it by osmosis without needing someone to tell them about it. The greatest cultures even transcend their own companies, and their own era, jumping from generation to generation. The biggest example of this is how even current-day Silicon Valley culture is very close to the that of Fairchild Semiconductor, which in turn was largely the creation of a single person–Robert Noyce.

Interviewers and candidates often talk about “culture fit”. This is a negative filter. Interviewers aren’t looking for people who are just like them–most reasonable people should be able to inculcate a reasonable culture just fine–they’re looking to weed out people who are inflexible or otherwise indoctrinated enough to reject the learning-by-osmosis that takes place when one first joins an organization.

But here’s the thing: amorphous as it is, culture makes things happen. It does have a real impact on how products are built, and which ones are built. I like to think of it as the bass player in the band. They are the most underrated member of the band. But when there’s no bass, the whole thing sounds off somehow, and you can’t put a finger on what exactly is missing. Culture is like that. Its presence in a place is taken for granted until you go somewhere else where the culture is different, or missing. It took me a while to figure this out. And then I realized why a select circle of seniors (in accomplishment, not necessarily in age) were constantly tending to it.

Which makes it all the more important to get a better handle on what exactly it is.

For engineers, there are two types of culture. The culture of the company as a whole, and the culture of the engineering organization within it. In true tech companies, the two are essentially the same. The entire company’s culture is ultimately dictated by what makes it tick. Some companies are sales driven. Others are product driven. And some are truly tech/engg driven, where cool things are built by smart people, and then someone thinks about how to make money off it. Don’t jump to the conclusion that the last one is great. The purest example of that was PARC, and while it gave the world the defining elements of the computing age, the entity itself could never stay financially solvent.

And then there’s engineering culture. It turns out, this is much simpler to pin down, but there’s so much fog around the world “culture” that it gets lost.

The engineering culture is simply the set of behaviors positively reinforced by the tools and processes of the organization.

If code review is a pre-requisite for submitting patches, then you have a culture that encourages clean code, early bug-catching and knowledge-sharing. If code reviewers insist on tests for new code, and there are tools that make it easy and fast, and documents that explain clearly how to do it, then you have a culture where testing is ingrained. Note how self-reinforcing things like this are. If your code was put through the scrubber by reviewers, you too will do the same with code that you review. And that’s a good thing. That’s what keeps standards high. This is beautifully meta too—the culture is defined by software, in a sense.

I was in two minds about how much of the engineering tools of Google I could describe in detail, but it turns out ex-Googler Mike Bland has written a series of five posts that goes into the coding and testing culture and tools at Google in excrutiating depth. It is a splendid case study of how engineering culture is established, changed and propagated.

A teaser:

This is the fifth—and, finally, final—post in my “whaling” series about the high-level conceptual and cultural challenges the Testing Grouplet and its allies faced, and the knowledge and tools that eventually spread throughout Google engineering that removed the infamous “I don’t have time to test” excuse. This post discusses the specific tools the the Testing Grouplet, Testing Tech, Build Tools and others developed to improve development and testing efficiency and effectiveness.

The first post in this series focused on high-level cultural challenges to the adoption of automated developer testing that emerged as a result of day-to-day development reality. The second post in this series focused on the fundamental object-oriented programming issues which formed the core of most of Google’s testability challenges—and solutions. The third post in this series covered the basics of how automated tests should—and should not—be written. The fourth post in this series described the collection of processes employed by Google for ensuring software quality—including, but not limited to, automated testing.

Which means it will probably take most people by surprise.

He starts by building the case for “follow your passion” being bad advice. Instead, one should focus on becoming really good at something valuable. Hence the title. True job and life satisfaction, he argues, comes from having options and flexibility, which in turn comes from the career capital one builds up by continuing to improve and being valuable.

The rest of the book is advice about how exactly to do that. But whether one follows that advice or not, the mental realignment offered by those first two theses alone is worth the price.

Career advice books usually massage your ego. Yes, you are unique and great. You must believe in yourself. It is just a matter of time before you strike gold!

This book gives it to you straight. You have to get good before you get good work. There are no shortcuts.

Highly recommended.

Part of it is simply email overload fatigue. The promise of email has been subverted by a number of factors:

• it is no longer asynchronous: Yes, in theory email’s biggest advantage is that it is asynchronous. I send you a message, it sits in your inbox and you reply to me at your leisure. In the meanwhile, both you and I go about our work merrily. This is simply not the case anymore. People expect a reply pretty much immediately, or within a few hours tops. And we–you and I–the ones who jump on every email and reply to it right now are to blame. Email has completely lost its asynchronous nature, and now sits just a tiny hop away from instant messaging in terms of immediacy.
• email can’t be deep: it is simply the wrong medium for deep, weighty discussions. Just because you spent an hour pouring out a thousand word essay into an email doesn’t mean the person at the other end with a four-digit number in a red circle showing his unread count will spend the same amount of psychic energy on it. Email is for notifications: “take a look at this”, “hey, this thing broke”. It is for skimming, not for diving. We all know the exponential loss of signal by the time an email thread crosses ten replies or as many people in the to/cc lines.
• it is inherently attention-fracturing: An inbox is like being in the center of a circle of people, each repeatedly shouting the same phrase at you. Your best bet is to either knock them out (unread archive), or make a quick radial sweek (next, next, next). It is not the place to go for shoring up your attention and focusing.

And it is exactly for all these reasons that shared documents shine.

• Writing a document focuses you and structures your thought in ways email cannot. It makes you think deeper, further out. The same goes for your readers. Reading well thought out prose will elicit more meaningful feedback than email ever could.
• comments are inline: in one place, right next to the main body, linked to the part that is commented upon. When you open your document, all the discussion is in one place.
• all the privacy of email is retained because docs can be shared with just the people you want, or nobody, or made public if you want.
• And last but not least, with a document you have an artifact that you can refer and link to as a record of work done and thought recorded, as opposed to an email thread.

For the longest time, it was thought that functional programming would save us from the multicore crisis, because, well, if there is no shared state then every concurrent execution can just blaze ahead full speed, can’t it? But the devil really is in the details. Persistent data structures are not always performant. Sometimes it is cheaper to update in-place (Eeek! Assignment!).

Also, as Cedric Beust points out, in the meanwhile the imperative camp did a great job of making concurrency work, by putting out libraries that solved the hard problems and yet exposed easy enough APIs. Paradigms win and lose not just by their theoretical beauty but by how much engineering effort goes into making them viable. As an example, this is what it takes to get a high-performance concurrent dictionary.

But all the same, the functional camp has much to be proud of. The trend of mainstream languages incorporating functional features has gained a lot of momentum and shows no signs of reversing. Anonymous lambda functions. First-class functions. List comprehensions. A decade ago they were FP-only, and now they are part of most programmers’ vocabularies. Even C++ has lambda functions now! Maybe all languages are destined to asymptotically approach Lisp.

We’re seeing functional-inspired features creep into mainstream imperative object-oriented languages. Richard Minerich calls it expression-oriented programming. I half-jokingly call it fimperative programming. But what’s driving that isn’t so much the need for performance or dealing with multi-core, but just more succint and cleaner code.

Sidenote: while functional programming is often presented as the antithesis of the now dominant object-oriented model, immutability and getting rid of state (i.e. no assignment!) were central to Alan Kay’s early ideas of what objects should be like.

Does technology induce feelings of gratitude in you? Does it make you behave better? Do you ingest a life-enhancing (or life-saving) drug and then go about your daily snark? When your browser loads a webpage do you marvel at the straight line made through a stack of disparate but amazing technologies (silicon, displays, networks, Ethernet, TCP/IP, HTTP, HTML, CSS, Javascript…)?

I suspect the answer is no. And at some level that always upsets engineers. We want people to feel the same sense of awe at our creations as they do when they see oceans and mountains. To feel humble and repent their petty ways.

We want the world to be like Star Trek. People are clean and reasonable and polite and well-behaved. Violence and conflict are moderated by thoughtful leaders (Kirk, Picard). Human emotions are finely embroidered into the plot, not splashed randomly.

But the real world is a lot like another lesser known sci-fi series — Babylon 5. It shows a world with spacefaring humans, but also with politics and class and violence and addiction.

Technology doesn’t change human nature.

In terms of hard-nosed numbers, the key statistic I’m pleased about is that RSS subscribers grew 3.3X over the course of this year.

These were the most popular posts of the year:

• Productivity Porn
• What is life like for PhDs in Computer Science who go into industry?
• Why you would want to program at fifty, or any other age
• Advice to prospective grad students
• English is heavily left-handed
• Teaching data structures with real-world examples
• My setup
• New frontiers in text editing

Please let me know if something looks bad on some platform.

And I chomped through more books this year than probably the last five. The physicality of books, sensuous and nostalgic as it was, was actually holding me back from reading. People who lament the death of the physical book due to the onslaught of e-readers, I’m sorry, but you have zero sympathy from me. The book might by dying, but it is the greatest time ever for reading.

So here are some of my more memorable things I read in 2012.

Fiction: Very little of what I read was fiction. I found some short stories online which were “sent to the Kindle” (or the Pocket app) for clean reading. The ones which stand out:

1. “The Paper Menagerie” by Ken Liu: An absolutely heartbreaking short story about a boy born in America of immigrant parents, whose culture and ways seem alien to him. I can see why it swept all three of SF’s major awards: Hugo, Nebula and World Fantasy.
2. “Mr. Penumbra’s 24-Hour Bookstore”, by Robin Sloan: the short story that the author later expanded into a novel. All I will say is that it is the story of a bibliophile’s adventure.
3. “Microserfs”, by Douglas Coupland: a short story that is more than a decade old, published in Wired back in the day. I re-read it again recently, and it brought back not just the story itself, but also who and where I was at the time I first read it. Nostalgia is heady. An unadorned portrait of a lonely hacker looking for solace in machines, and maybe, people. I know this will ring true to many of us.

Non-fiction: Looking back, almost all of it was about the nature of modern work in one way or another. This is a topic that has drawn me in. I think it started a few years ago when I read Matt Crawford’s “Shop Class as Soulcraft”.

1. “The New American Dream”, by Penelope Trunk: Using interestingness as a sniff-test for where to go in life and career, and how to strive for that ever elusive work-life balance. I bought this book simply because I’m a long-time reader of the author’s blog, and a fan of her voice. And it turned out to be a great read.
2. “So Good They Can’t Ignore You”, by Cal Newport: Another book I bought right away because of the author’s blog. (Do you see a pattern here? These days, I don’t even consider buying a book if the author doesn’t have a long trail of words online.) Makes the case for “follow your passion” being bad advice, and that you have to get good first before you get good work.
3. “The Computer Boys Take Over”, by Nathan Ensmenger: Delves into the history of programming computers as a craft and a profession. Many current issues about the profession of programming have deep historical roots, and this book gave me a lot of perspective. (My review.)

I’ve also noticed the new gesture typing feature in jellybean make input so much faster. It is pretty forgiving, and has a high rate of guessing the correct word even with seemingly random scrawls on the keyboard.

I’ve written the these pieces on the tablet. Reading them back, I can see a difference. There is a brevity and tautness to them. The tone might even be somewhat stern.

On the other hand, this piece was dictated out, and you can certainly see that I’m freer with words and have a more conversational tone there.

If you are a writer of fiction, or a writer of non fiction who has all their research and notes at hand, then yes, in theory doing what you need to do becomes a battle of will power where you just need to sit still and concentrate and get the words out.

But most knowledge work (I despise that term, buy I’ll use it just because it’s familiar to people) is fragmented by nature. I like to think about it in terms of what I call the quantum of work.

A quantum of work is the theoretical longest amount of time you can work purely on your own without needing to break out into looking up something on the web or your mail or needing input from another person. For most modern workers this quantum of work is measured in minutes.

What’s worse is that sometimes when you break out of a quantum you have to wait to get a response. What are you going to do in the meanwhile? You start working on another thread of work to utilise your time. You keep doing this until you get the response you were waiting for, and get back to the first thread.

Is there a way around this? To work in an uninterrupted manner you have to end up changing the nature of the work.

Is that possible? I’m not sure.

The trajectory from linear long form work to fragmented quantized work had been quite steady and there are absolutely no signs of it reversing.

That is horrible advice.

Before I lay out my disagreement, please take what I say with a huge grain of salt. Marco’s hacking credentials are impeccable. He was an early (the first?) programmer at Tumblr, which is serving up these very words to you, and has single-handedly built some of the finest mobile apps for people who appreciate pure reading (like me). He’s clearly doing something right.

But it is those very credentials that give his advice great weight, and which, in this instance, is dangerous. If indeed he built complex apps like Instapaper without a whiff of automated testing, that means he’s superhuman. Just because Bolt can run 100 meters in under 10 seconds doesn’t mean we can all do it.

If there’s one process-related lesson we should have all learned from the last thirty years of programming, it is this: automated testing pays off.

There’s hard data to back this up. Test-driven development reduces defects drastically, while not hurting development time.

The terminology is varied: unit-testing, functional testing, integration testing etc. The important thing is that it be automated, and to a lesser extent, quick. Something you can run quickly (preferably with a key combination) every time you make even a small change, to make sure you didn’t break anything. And write new tests to cover any new code you’re writing. What else are you going to do? Run through every functionality in your app manually to make sure everything is still OK? How long do you think that will take? Why? When you could run a script?

Having automated tests increases your velocity, because you can make changes with more confidence. It can become a competitive advantage if done right. And it is worthwhile even in areas where the cost of defects is not high. My personal experience is that for anything that’s larger than a small self-itch-scratching script, that is expected to grow, the small price of writing tests is always worth it. And I’ve almost always seen my code get cleaner and more modular under the pressure of making them testable.

Please code like you’re in the 21st century. Write automated tests.

(After writing this I found that Joel Spolsky made more or less the same point here.)

But given that, an interesting question is how to determine what to commit to my personal wet memory, and what to store outboard in dry memory. Given some random fact, is there a test I can apply to it to determine where it should go? How many times have you thought that you should have remembered something without having to look it up, or that you really didn’t need to clutter your head with something and could have just as well relegated it to your outboard brain?

Imagine if it took you five seconds to remember your spouse’s name. Or even their birthday. That would be bad. So latency of access is an important criteria. You should remember things you need to know really fast.

But with things like augmented vision, and further out, brain implants, the latency difference between wet and dry memory will vanish.

Memories are cherished. They are the fodder of nostalgia. We go to great lengths to acquire them. How many dares have you done thinking it would make a good story to tell your grandkids? What exotic locales have you traveled to to make great memories?

But memories are also stressful. There are unpleasant things you don’t want to remember. There are things you need to remember that are stressing you out – birthdays, appointments, todo lists. GTD caught on like wildfire because it brought a workable system to bear on externalizing all that necessary trivia that makes up a modern life.

For much of human history, the default answer to “should I remember this?” was “yes”, with narrow and specific exceptions. Right now, we’re somewhere in the middle. But all the technological trends inevitably point in the direction of turning the default to “no”, with fewer and fewer exceptions.

Does this mean we will eventually turn into memoryless biomass? I don’t think so. Because memory is what makes us who we are. It makes us individuals. Ultimately, we won’t need to remember anything, so we will only remember what we want. It will be the purest expression of ourselves.

The question was: “Should I take and internship with Microsoft or Amazon?”

Disclaimer: I haven’t worked at either company. This is a more general answer.

First of all, congratulations! You’re in the enviable situation of not going badly wrong either way.

They’re both decent companies working on challenging problems. And I’m assuming both offers are in the Seattle/Redmond region, so geography is not a differentiator.

You should choose based on two things:

1. Did you get to speak to the person who will be hosting you, i.e. your boss for the summer? Do they seem like someone who would take the time and effort to guide and mentor you? Are they enthusiastic about what they’re working on? About having you? Are they technical or more managerial? If your official host is a manager and will hand you off to one of the engineers on their team, ask to speak with that person. I’ve hosted interns myself, and it takes special care to see to it that they are successful, and are learning and enjoying their time there. One has to pick problems of just the right size and challenge. You want to feel like you accomplished and finished a decent challenging chunk of work over the summer. An internship is basically bootcamp for industrial software engineering, and you want a great coach.

2. A distant second factor is what exactly you will be working on, and whether it is something you are interested in. I say distant second because you should not get hung up about trying something outside your specialty or interest. The main point of an internship is to just see how software is built in the real world, and get to contribute to it in a real, non-toy way.

Hope that helps!

Consider a totally toy example: print out the squares of numbers from 0 to 9. In “old” C++ this would look something like:

#include <iostream>
#include <list>
#include <algorithm>

using namespace std;

int square(int i) {
  return i * i;
}

int main() {
  list<int> l;
  for (int i = 0; i < 10; i++) {
    l.push_back(i);
  }

  list<int> t;
  t.resize(l.size());
  transform(l.begin(), l.end(), t.begin(), square);

  for (list<int>::iterator it = t.begin(); it != t.end(); ++it) {
    cout << " " << *it;
  }

return 0;
}

Using the new C++11 features (Range-based for loops! Lambda expressions! List initializers! Type inference!), that becomes somewhat shorter and cleaner, while not sacrificing an ounce of its grip on the bare metal:

#include <iostream>
#include <list>
#include <algorithm>

using namespace std;

int main() {
  list<int> l = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};

  list<int> t;
  t.resize(l.size());
  transform(l.begin(), l.end(), t.begin(),
        [](int i){ return i * i;});

  for (auto e : t) {
    cout << " " << e;
  }

  return 0;
}

Of course, detractors will say that none of that comes even close to the succint Python solution:

l = range(10)
t = [ x * x for x in l ]
for e in t:
  print e, ' ',

Note that a big win in terms of expressivity in the Python solution comes from list comprehensions. Perhaps the next C++?

So I came back to that during the downtime this Thanksgiving, and brought the copy up to date. I think it’s a good way to quickly thumb through the archives.

The tribe of those who make their living by writing code has a lot to be thankful for. Here are some things, in no particular order.

(For the pedantic among you, yes, there are good days and bad days. Just prepend the phrase “by and large” to each of the following.)

• There is a lot of work to be had. As with some other professions, we do not have to worry (yet) about a shrinking field.
• It pays well enough to keep us on the higher rungs of Maslow’s pyramid.
• We get to work with smart, enthusiastic, reasonable and passionate people.
• We get to work on something that is intellectually stimulating.
• Our work does not feel like work at all.
• The tools and artifacts we make (the biggest example is the Web) fold around to become our creative and expressive outlets.
• We build artifacts that are used by, and improve the lives of, many people.

If you look around today, there are very few professions for which all of the above are true (and I think the number is shrinking).

For all this, I am deeply grateful.

But please don’t do that. That is the one thing that will instantly poison your chances. If you think the question is trivial, prove it by banging out a clean, correct solution, quickly and without fuss. Show them how easy it was for you. Then you might get asked more challenging and open-ended questions.

@slava_pestov @vivekhaldar What if you know how ill-conceived that interview technique is, and it makes you lose interest in working there?

— Richard Feldman (@rtfeldman) November 21, 2012

Think about it from the point of view of both the company and the interviewer. For the company, the cost of incorrectly hiring a moron is much, much higher than incorrectly passing on a genius. And please believe me when I tell you that that simple question you thought was beneath you stumps a good fraction of candidates.

These two factors combined mean that interviews have largely become negative filters, i.e. they try to filter out people below a limit, rather than trying to gauge how bright you shine. But good interviewers will almost certainly kick the gears up if you make easy work out their initial questions.

I realise it is far from perfect, but given the constraints, it’s probably the best we have.

By far the best “interview” is to have the candidate on-site for a while and have them work with a team. In other words, an internship. That’s why a lot of time and effort goes into internship programs. Killing it during your internship greatly increases your chances of getting a full time offer.

It’s high time somone called bullcrap on this meme.

That’s akin to saying that the purpose of an aircraft carrier is to keep afloat the flag at the top of the mast.

“Clicking on ads” happens to be one single application of a lego-like complex of massively complicated and cutting edge technologies that underlie a whole host of other applications. These underlying technologies span the gamut of creating and managing warehouse-scale computers, massively distributed storage systems, and manipulating large amounts of data for machine learning techniques. This involves solving deep and challenging problems both at the level of science, and at the level of engineering to make it real. It it used in multitudes of applications beyond ads.

So please, look deeper at the general problems being tackled, and their myriad applications.

Like most articles that bemoan the oncoming death of the university, that article suffers from the logical flaw that it conflates the two components of a university: teaching, and research.

MOOCs will almost certainly hollow out the teaching component of universities as it stands today. I don’t see anything on the horizon that will reverse this tide. In most technical fields, the nuts and bolts technical interview and on-the-job learning and performance monitoring long ago replaced any faith in degrees as credentials. That leaves very few fields, such as law, where you absolutely do need the degree as a credential.

Add to that the fact that instructors of MOOCs feel much higher levels of job satisfaction, because of the impact of teaching a few thousand students, many of whom may not be privileged enough to get to a physical classroom. Contrast that with teaching a class with tens (or for large introductory courses, hundreds) of students, most of whom are surfing on their laptops.

Unfortunately, teaching is also the component that brings in the lion’s share of money into academia. But all is not lost, because the other thing universities do is research, and that is arguably as important, if not more, than teaching. If universities shed their easy-money-making credentialing and teaching component, and contract to becoming purely centers for research, that might not be such a bad outcome.

On the tradeoff between growing the language to support complex (but necessary) functionality, and keeping it simple:

Early on in TinyOS development we made the decision to design a language to better support its pro- gramming and concurrency model. The nesC language allowed TinyOS to achieve near-optimal resource effi- ciency (minimization) and a surprisingly low bug rate (prevention). Having a new language also allowed us to evolve and extend features as new problems arose. Language evolution is a two-edged sword… each stage of this evolution added new features, moving TinyOS and nesC further from C and raising the barrier to entry. Furthermore, the most effective software patterns, such as static virtualization, used all of these features in complex and novel ways. By focusing on expert TinyOS users and making it possible to write larger software, TinyOS 2.x became less accessible to new users. Making it harder to write buggy code had the unfortunate result of making it just plain harder to write code.

On the importance of quick rewards for beginner users:

In retrospect, the focus on expert users missed a great opportunity: hobbyists and the “Maker” do-it-yourself crowd…In comparison to TinyOS, Arduino is feature-poor: programs are single- threaded C programs for simple sensing and actuation. But for hobbyists, the resulting simplicity is extremely desirable. Building a gumball machine that “only dispenses treats when you knock the secret rhythm on its front panel” (an article in a recent issue of Make maga- zine) doesn’t require static virtualization, network types, and compile-time data race detection.

And on that long-contested question in software engineering, the usefulness of reusable components, and how hard it is to actually achieve that²³. TinyOS has a fine-grained component model, but it turned out not to help.

The main goal of this approach was to ease experimentation. But taken to its conclusion, fine-grained components have significant drawbacks we did not foresee…the drawback of fine-grained components emerges when trying to understand a system for the first time. There are so many tiny pieces of functionality spread across files, with numerous levels of indirection, that keeping track of it all can be a headache. The structural complexity is far beyond what the underlying code complexity requires. Our conclusion is that a well designed and carefully implemented operating system is more helpful than an operating system toolkit or operating system software designed with reuse in mind. Our experience with deveoping more traditional operating systems supports this conclusion. It is easier to take the Linux boot code and modify it for your needs than to work within a component framework for its generalized boot module

(All emphasis mine.)

Let me explain what each of those means to me.

By deep history, I mean having very long lasting shell sessions, so that I can go back and see the entire history of my interaction (commands I entered as well as their output) going back not just days but months.

By “an unrestricted, editable shell”, I mean a session where everything is editable and searchable, not just the command prompt. In other words, your shell is just like a plain text file in an editor.

So some of you have probably already guessed that I implement this using shell buffers in Emacs. Within that, I have tons of shell buffers, usually one per task or project I’m dealing with, all with short but descriptive names. In addition I always have Emacs running in daemon mode. Almost always, the uptime for my Emacs process is the same as the uptime for my machine.

This combination is extremely powerful. I don’t know how many times I have to recall some complex, obscure commandline invocation, or remember some piece of interaction from the past. With this setup, all I have to do is search backwards. Also, selecting, copying and pasting text in most terminal emulators is is very fiddly, and usually involves leaving the keyboard and using the mouse.

The bottom line is that each shell is just another plaintext buffer, upon which the full force of Emacs’ text editing power can be applied. It now makes me cringe when I see someone fiddling with tons of little unsearchable, uneditable xterm windows, struggling to copy and paste text in and out of them.

There is a downside though. On the rare occasion that my Emacs daemon crashes and I lose all my state, it is extremely disorienting. Those are the chips.

Whether you want to do something at an arbitrary time in the future is a superficial, if interesting, question. The deeper question is what motivates you do it now, and how you think that motivation might shift (or maybe altogether disappear?) in the future.

So, no, I honestly don’t know what activities will motivate and excite me when I am fifty, just as I didn’t know at fifteen what would make me get out of bed this morning.

I have seen, and personally felt, a range of motivations for programming. Some of the more common ones are:

• Money. It’s a well-paid job. It pays the bills, and leaves over fun money. You don’t mind it.
• Purpose. Your programs help a lot people do their jobs, live and enjoy their lives.
• Creator’s machismo. Your programs handle a gazillion queries per second and run on a trillion machines and make a jintillion dollars every minute. Take that!
• Beauty. You find programming artifacts to be beautiful objects in their own right. You use it as a channel for self-expression, just like sculptors and painters. You enjoy the feel and flow of your tools and environment, just like some enjoy brushes and palettes and chisels.
• Intellectual exercise. You find programming to be a grand intellectual adventure. Not only is it fun and challenging, but gives wings to your thoughts by making them executable. You use computation to explore the boundaries of the possible. It staves away boredom.
• You fell into it by accident or circumstance, and one or more of the above compelled you to stay, at least for a while.

Of course, most programmers will cite more than one of the above as their motivations, and smear it with their own weights. What are your picks, and the weights associated with them? How do you think they will be different a few years down the line?

How much can one factor swing negatively while still making the overall proposition desirable? Would you still enjoy it if your programs had no intellectual heft but did an important and necessary job? What would you look for if you had saved or made a stash of money and didn’t have to worry about that?

This is ultimately a question which can only be answered by looking inwards. It is a test of self-knowledge. You don’t match your age to your job. You match your motivations to your job.

What struck me was how specific and nit-picky the feedback was. For example:

We sat in the surgeons’ lounge afterward. He saw only small things, he said, but, if I were trying to keep a problem from happening even once in my next hundred operations, it’s the small things I had to worry about. He noticed that I’d positioned and draped the patient perfectly for me, standing on his left side, but not for anyone else. The draping hemmed in the surgical assistant across the table on the patient’s right side, restricting his left arm, and hampering his ability to pull the wound upward. At one point in the operation, we found ourselves struggling to see up high enough in the neck on that side. The draping also pushed the medical student off to the surgical assistant’s right, where he couldn’t help at all. I should have made more room to the left, which would have allowed the student to hold the retractor and freed the surgical assistant’s left hand.

Code reviews also tend to be nit-picky. Style issues, formatting, naming. But what seems to be nit-picky ends up being important. When dealing with very large repositories of code, these seemingly insignificant flaws are like pebbles on the tarmac. If you let them build up, pretty soon your smooth tarmac becomes a gravel road.

My own “aha” moment for realizing the true value of code review came to me when I noticed that I myself was writing the kinds of messy code that I would call out when reviwing other people’s code. But I was blind to it because I wanted to quickly get to the point where my damn code would work already. We all are.

Age

Flakiness in software is most noticeable when it is in the foundational layers – the operating system, basic system services such as those for networking etc. And it is exactly those foundational pieces that by now have had two decades or more of continuous existence. This means they have been exercised and tested in every way imaginable in the real world. It also helps that the responsibilities of almost all foundational infrastructure software has barely changed during that same time.

For example, what an operating system kernel does has not changed in a very long time. The basic networking protocols – TCP/IP, DNS, HTTP and many others – have also been remarkably stable, with mostly incremental improvements. The sheer weight of everything else built on top of this layer has turned these into diamonds.

Server-side software

Most software that is today used by end consumers, i.e. clients, is but a thin pretty shell backed by an array of services running on a large collection of machines off in a data center somewhere. This has several implications for the overall perception of reliability.

• server-side software is tended by humans. I mean that literally. When error rates go up, humans are woken up by alerts, and they deal with the problem. In most cases, the issue will be taken care of without any significant disruption, or anybody even noticing. For every major outage where a site or service is completely down, there are a thousand such problems.
• server-side software can be updated under controlled conditions, on demand.
• embarrassingly parallel loads spread across a large number of machines means a high level of aggregate reliability for the entire service.

In the era of heavy client side software, these would all be major disruptions.

In essence, the move to server side software allows us to cheat when it comes to reliability. You can deal with problems as they arise. You don’t have to worry about a large existing installation out in the field. This also means you don’t need to front-load hardening and testing.

White is what I call a primitivist. Not really a Luddite, as in someone who dislikes and opposes technology, but someone who is skeptical of it, and prefers the simple “human” solution to the adorned technological one. Hence the part where he fashions an electric guitar out of a few old pieces of wood, a few nails and a single wire. Raw is better than cooked. Less is more.

Look at this performance by Son House, one of White’s idols. Any musician will immediately point out that the damn guitar is out of tune. They will also concede that, in this case, it does not matter. That one man with one impaired instrument is more than the sum of the parts. The emotion and gravitas of this piece put today’s studio engineered sounds seem as sterile and cold and gray as a freshly scrubbed hospital hallway. Look at his weathered face, his calloused and stiff fingers. Only a man like that can pour out a song like that.

Page and The Edge are closer to the other end of the spectrum. Their musical explorations were also very much an exploration of the boundaries of technology and how much they could push it. Amps, pedals, effects, fancy guitars, sound engineering. They were not shy about using any of those tools to express what they wanted to express acoustically.

Look at this clip where he plays the intro to Elevation, with and without the effects pedal.

Like he says in the documentary, he is just trying to get the machinery to make the sound he can already hear in his head. So obviously, the technophiles can also emote and express. They just have a very different way of going about it.

Overall, fascinating documentary, and a must-see if you are even a little bit into rock. The money shot is the three of them jamming in the end. You can watch the whole thing on Youtube.

And then reality hit. They announced a price of £199, which at today’s conversion rate works out to approximately $320. Love it as I may, that was too much for a printer. For comparison, the cool new Nexus 7 was only $199.

And thus was born my weekend DIY project to build me my own little printer.

These were the ingredients:

• some cheap thermal point-of-sale (POS) printer I found on Amazon. $51
• pack of 24 rolls of 2.25 inch, 85 ft thermal paper. $9
• Python code for outputting to POS printers.
• My own little scripts for fetching RSS feeds and outputting them in various formats, now updated to output ESC-POS format.

Total money spent: $60.

Because I was doing this on my MacBook and I had trouble finding the Python USB libs, I had to fork and hack the Python escpos code to output to files instead of sending directly to USB. This also let me decouple the fetch/format stage from the actual printing.

On MacOS X, install it as a generic printer, and then print to it using lpr. The scripts will output files in the ESC-POS format, which you can send directly to the printer:

lpr -P posprinter -l nytpic.prn

Now this is certainly not as polished and convenient as what Berg is offering, but at less than one-fifth of the price, I humbly contend that it gets you much more than one-fifth of the way there.

Now I’m certainly not trying to say that money is not important, and that it should not be discussed. Everybody needs to make a living. Everybody needs to feel secure. But what the tech press is missing is the entirely different attitude that true Valley entrepreneurs have towards money.

The Wall Street attitude towards money is to use it for two primary things: status, and insurance. As you get more money, you get to buy more expensive toys. You also have the secure feeling that no matter what problem you face, you can throw money at it to solve it.

But Silicon Valley is very different. First of all, there is almost no status hierarchy. Billionaires dress the same way that Ramen-eating fresh graduates do – in well-worn jeans and t-shirts. Fresh graduates address them by their first names, and have irreverent, spirited debates with them. Pulling rank is considered to be extremely poor form.

Money is treated differently. Its primary purpose is to be used as fuel. Fuel to get somewhere. Fuel to do new things. Fuel for an infinite game.

Here is Denning¹:

Science, engineering, and mathematics combine into a unique and potent blend in our field. Some of our activities are primarily science—for example, experimental algorithms, experimental computer science, and computational science. Some are primarily engineering—for example, design, development, software engineering, and computer engineering. Some are primarily mathematics—for example, computational complexity, mathematical software, and numerical analysis. But most are combinations.

But Vint Cerf, the recently appointed president of the Association for Computing Machinery, gets the last punch²:

In the physical world, science is largely about models, measurement, predictions, and validation. Our ability to predict likely outcomes based on models is fundamental to the most central notions of the scientific method. The term “computer science” raises expectations, at least to my mind, of an ability to define models and to make predictions about the behavior of computers and computing systems… When we write a piece of software, do we have the ability to predict how many mistakes we have made (that is, bugs)? Do we know how long it will take to find and fix them? Do we know how many new bugs our fixes will create? Can we say anything concrete about vulnerability? What about the probability of exploitation?

In other words, you can’t call your field a science until it gives you predictive power over your world.

At the beginning of a complex software project, the margin of error in predicting completion time and resources required is so large as to be practically useless. This cone of uncertainty decreases as the project progresses. The only way out is through.

Estimating (i.e. prediction) is the dirty secret of the computing industry. Everyone wants them. Everyone provides them. Nobody attaches probabilities to them. And absolutely nobody believes them. Reported estimates are often padded by a factor of two (or more) for planning downstream activities, and even that isn’t enough.

I am pessimistic about the endeavor of bringing scientific estimation to software development, at least, in an a priori manner. It might be possible to build post-hoc models based on history. For example, by the tenth time you build the same thing, you can provide a good estimate for the time and resources required.

I am pessimistic because the bulk of uncertainty in software development comes from human complexity. If the world and behavior of humans was governed by principled laws, and if they all used the same language, then software in such a world would be simple and predictable. But the world of humans is complex and messy and ureasonable, and software that deals with them also must be.

• there are strong incentives to reduce power consumption, as that is a huge capital drain, and, well, companies want to reduce their costs. (I found it strange that one of their sources was quoted as saying that they don’t get bonuses for reducing power consumption.)
• each machine in a datacenter serves many people. That desktop serves just you, and that too for a small fraction of the day. A world in which everyone uses tiny devices (phones, tablets, laptops) backed by giant datacenters will use drastically lesser power per person than one in which most have desktop machines.

The original study on which the claim is based claims a relation between assset allocation and volatility, not returns.

This paper¹ clarifies:

The specific claims vary, but financial professionals generally assert that asset allocation is the most important determinant of returns, accounting for more than 90 percent of performance. This assertion stems from studies by Brinson et al. (1986, 1991), that state, “…investment policy dominates investment strategy (market timing and security selection), explaining on average 93.6 percent of the variation in total plan return.” This conclusion has caused a great deal of confusion in both the academic and financial communities. In fact, a survey by Nuttall & Nuttall (1998) demonstrates that out of 50 writers who quoted Brinson, only one quoted him correctly. Approximately 37 writers misinterpreted Brinson’s work as an answer to the question, “What percent of total return is explained by asset allocation policy?” and five writers misconstrued the Brinson conclusion as an answer to the question, “What is the impact of choosing one asset allocation over another?”

(emphasis mine)

The depressing lesson to be learned from this (and countless other times I’ve dug into the primary source behind a citation) is that citations of technical material in popular literature simply cannot be trusted. If you really want to know, go find the paper and read it. Citations of technical material in technical literature fare better, but even there results are often mis-interpreted or cherry-picked to suit one’s position.

As they say, use the force, read the source.

(So how much of returns are due to asset allocation? This paper² might answer that question.)

What I don’t understand is: why should you ever care how your editor looks, unless you’re trying to win a screenshot competition? The primary factor in looking good should be the choice of a good font at a comfortable size, and a syntax coloring theme that you like. And that is not something specific to an editor. Editors like Emacs and vi have almost no UI! If Emacs is configured right, the only UI it has is the modeline and the minibuffer.

People talk about getting used to a new editor, but over time, it is precisely the opposite that should happen–the editor should get used to us.

I watched this drama from my perch upon Mt. Emacs. If this makes me sound like an Emacs elitist snob, so be it! Now, I’m typing these words into Byword on a Mac. For me, it serves as the small home kitchen knife compared to the full chef’s knife set that is Emacs.

There are times when I just need to serially dump a large number of words from my head into the digital ether, and that does not require much editor sophistication. There are other times (the majority of my day job) when I need juggling between dozens of files and shells and compiles and regexes and searches and syntax comprehension. That’s when I use Emacs.

Overall, I think text editing is a solved problem. Text editors are like wine. The older the better. You do not want a shiny new text editor. You want the text editor that has been around long enough and used by enough geeks that every conceivable pattern of manipulating symbols on the screen has been thought of, and crystallized into a re-usable pack.

The big unsolved problem in text editing is to do it without a keyboard, whose use I think peaked sometime in 2011. On tablets and phones. On devices where the entry and manipulation of text remains a royal pain, because we haven’t yet figured out a way to do it. The challenge is to never make someone think “damn, I really wish I had a keyboard right now!”

If you go by the relative frequencies of the letters in the English language, and assume a keyboard split like the Goldtouch keyboard, it turns out that characters on the left occur 58.73% of the time, and those on the right 41.3% of the time. This is certainly not helped by the fact that the right side has 11 letters to the left’s 15.

Add to that the fact that if you’re an Emacs user (like me), then some of the most common commands and command prefixes (Ctrl-A, Ctrl-E, Ctrl-X, Meta-X, Ctrl-C, Ctrl-R, Ctrl-W) all use left-side keystrokes, and you end up with an even heavier leftwards tilt.

Those who write know that it makes them happy. They know they will get cranky if they don’t. Just like athletes missing a training session. Now here is some emperical evidence along those lines.

• those who regularly wrote expressively before an exam had higher scores and lesser pre-exam depression. ¹
• writing about your worst experiences improves your mental and physical health. The flip side was that writing and analyzing positive experiences descreased overall life satisfaction. In other words, face and work through the bad stuff, accept and don’t overthink the good stuff. ²

The work of engineers

Start by asking yourself what you signed up for when you said you wanted to become an engineer, of software or any other kind.

Engineers are supposed to build real-world solutions to real-world problems within externally specified time and financial constraints.

This often means using stable, established technologies. Mature engineers understand that they have a limited risk budget. Almost always, that risk budget gets used up by uncertainties in time and money, which leaves little or no risk to be spent trying out cool and new, but unproved, technologies.

Out of time, money and experimenting with cool new stuff, they get to choose any two. This is why when a project that wants to push the envelope of the underlying technologies themselves, that is made (or at least, should be) explicit at the outset, and they are given a lot of slack with either money or time.

Maintenance

The narrative of technology is told in terms of invention and innovation. But if you look closer, innovation is made up of a thousand small, incremental steps on existing technologies.

Others have argued that looking at the history of technology through a use-based rather than innovation-based narrative paints a more accurate portrait of how prevalent and significant old technologies are, and what a central role maintenance plays. Things would simply fall apart if they were not maintained.

New Stuff, Interesting Stuff

Unfortunately, the dominant narrative of the progress of technology is the chronocentric one centered on innovation. That’s not a big surprise, because news needs to be, well, new. Everything reported in the tech news is significant and useful to a vanishingly small fraction of the population. The overwhelming majority uses boring, old, mature, dependable technologies.

The side-effect of this is that engineering graduates come into the job market with an expectation of always working on shiny, new, cutting-edge, greenfield projects. And then they’re dejected when that expectation is not met by the real world. There is simply not that much new stuff to go around.

But it is a mistake to think that working on an existing system, or maintaining it, is a crappy job. For a fresh grad, it might actually be a good thing. It is a gentle ramp up. It gives them a chance to observe the operation of a real system. If they take it upon themselves to dig deeper and learn about it and see why it was built the way it was built (as opposed to complaining about it), they can learn some valuable lessons. And if what they learn truly does not make sense, they’ve just found an opportunity for improving things!

We hire contractors and interior decorators to build our homes.

We hire accountants to take care of our taxes and investments.

We hire personal trainers to help with our physical fitness.

Why do we not hire software contractors (or whatever you want to call them) for specialized software work that is personalized to one’s specific workflows?

Programmers constantly build their own little tools and scripts to grease their daily workflows. Regular people are stuck with whatever applications happen to allow. If an application to do something does not exist, it usually precludes that activity entirely. Programmers don’t realize what a problem this is. Just listen to the convoluted flows on a show like Mac Power Users. A huge amount of time, money and effort is spent on cobbling together blunt tools.

How would one go about establishing such a profession?

But, first of all, come on, Andy, let’s not be ageist. One of the things he hates is:

Old people: Agile has the smell of death on it. If you go to an “agile” event you will see few people under the age of 40 and many over 50. These attendees are on average much older than the average age of programmers

It is not logical to dump on a state you will inevitably be in, i.e old. You might very well learn a thing or two from those that smell of death.

That said, I want to talk about what I like about agile.

Agile means so many things to so many people that is hard to even define what it means. I’m not going to attempt a universal definition. I’m going to talk about what “agile” means to me.

Here’s my definition:

To be agile means to have such a degree of control and understanding of the process by which you produce that you can make your output predictable (in terms of resources like time, people, material etc.) and can rapidly adjust the process to deal with changes in the environment (such as new requirements or contstraints).

For those who have a deep understanding of what it is, and how to implement it in a team setting, it works amazingly well. Like anything that works amazingly well, people want a piece of it. Like anything that works amazingly well, other people want to sell it. Hence the never-ending stream of books and hoopla and conferences and terminology.

Unfortunately, agile is often sold as a canned collection of processes and practices, and many think that blindly applying it to their context will make all their problems go away. No, it will not. If you just apply them like a script kiddie runs downloaded warez, without really understanding them or knowing how to tweak them, agile will not solve anything for you.

The same goes for agile “tools”, of which there are tons in the market. It is all too easy to fiddle with tools without getting any real results. It is not about the tools at all. I have seen experienced practitioners work wonders with nothing more than a simple shared spreadsheet.

Let’s look at the levels on the agile ladder.

1. The pain and the need. The team is somehow making stuff, but it is chaotic and stressful, dates and deliverables mean nothing. If deadlines are met it is by sheer luck. I need something, some process, to bring some control and predictability to this.

2. Lightweight process. Iteration. The next step is usually to wonder if a process can help you. Most people, especially technical folks, have a disdain for process, and rightly so, because a process can quickly become an end in itself if not overseen by someone mature enough to realize that it is just a tool.

   Fortunately, a lightweight process is all you need. This involves simple things like breaking work down into concrete chunks that can be estimated, fitting those chunks into timeboxed iterations (usually a few weeks), and people signing up for chunks in a given iteration. Just doing this much brings a lot of clarity to end-goals and the rough size of the problem being tackled.

   When you start out with this, you will realize that you aren’t very good at it. Your estimates will be way off, often by as much an order of magnitude. Your chunks of work may not be well defined. You will probably end up missing things you signed up for in the iteration, and doing things you didn’t sign up for. This is expected. Each iteration, keep an eye on what you could have done better. Stick with it. You’ll get better.

   The crucial idea here is that of iteration. You should try to get to a “walking skeleton” as soon as possible, and then add flesh to it every iteration. Failure should get exposed early rather than late. Which brings us to…

3. Visibility and automation. Next you get to what most people know of as the standard tools and practices of “agile.” Lots and lots of tests. Continuous builds and tests. Dashboards. Code metrics. There is a common theme underlying all these tools and practices–visibility and automation. Automation, because the more you automate, the more you remove friction and leave time to concentrate on the hard problems. Visibility, because the only way to even try to control a large, complex system is to gain visibility into it.

   You can get a big fraction of the benefits of agile by getting this far. I call this the “fake it until you make it” phase. Plan iterations (even if the plans turn out to be lousy), set up a continuous build, get used to being green and keeping it that way. You will notice the chaos and anxiety slowly but surely replaced by a calm, confident velocity. There should be far fewer fire drills.

4. Control and adaptibility. The thing that separates the 99th percentile of guru agile practitioners from the others is an understanding of your process and how to tweak it. What if your requirements suddenly change? What if you realize that you fundamentally need to rework some significant part of your product? What if your team size is halved? Or doubled? How will your process absorb these environmental changes and allow you to continue building and delivering stuff?

   There are no tools and practices for this. This just takes time and attention and gentle nudges in the right place.

   One principle is key here, and it is one almost never mentioned by agile tool vendors. That is the theory of constraints. That’s a fancy of way of saying that your process always has a bottleneck, and that you should find it and focus on removing that one bottleneck. Repeat with the next bottleneck.

   The two most common ways are to either put more resources to remove the bottleneck, or structure your work to avoid “backing up” at the bottleneck. For example, say you have ten devs and one tester. The rate of which your entire team can deliver reliable, tested code is limited by that one tester. You could either hire more testers, or start nudging your devs to test more.

That’s what agile means to me. Every team that gets value out of it uses their own unique variation of it, and that’s great. Take it on. Use what you like, throw away what you don’t. Pay attention to your team and your process. Don’t start by choosing tools. Take the word of agile “experts” with a grain of salt.

My initial exploration of the idea led me to develop what was a pretty simple and useless twitter chat bot (in the style of Eliza).

The next step in my exploration is Anagramr, a game of anagrams over Twitter. Here are the rules:

1. Follow @anagramr on Twitter. Every couple of minutes, it will tweet out a challenge like: “Give me anagrams for FOO.” (Update: I got feedback this was way too frequent. I’ve turned this down to one challenge tweeted out every half hour.)
2. Reply to one of those challenges with anagrams of the the word in the challenge. It must be a reply to that tweet, or Anagramr will not know which word you are replying to.
3. Anagramr will tweet you back with whether your reply was a valid anagram or not.
4. A couple of times a day, Anagramr will tweet out leaderboards.

A typical run looks like this:

Code is over on github.

The two layouts encourage very different styles of work, and even different styles of thought. In a recent study of a war room environment¹, extremely easy and frequent communication among team members was a huge advantage. The team in question delivered projects on time and on budget consistently.

But there were caveats.

You have to be a certain type of engineer to thrive in this kind of environment: “… [this] environment is not for every engineer nor for every design team. Certain types of personality, such as being flexible and adaptable, are needed to be able to work in such a public environment. Most work is visible to all (except intermediate calculations), especially mistakes. Team members unable to adapt to an unstructured environment do not last long on the team. Moreover, they have to be able to withstand the stress generated by noise and time pressure; despite these distractions, one must be able to continually monitor the conversations, exchanges, design status, and changing mission parameters, as well as the team leader’s remarks…. Almost all current Team X members report being mentally tired at the end of a session. Yet results from a questionnaire I administered to each team member showed the mean satisfaction level working in this environment to be quite high: 9.4 (sd=0.9) on a scale of 1 to 10 (high). One engineer described the experience as exhausting but thrilling, like riding a roller coaster. The key is to enjoy problem-solving in a highly social atmosphere.” (emphasis mine)

On the other hand, the frequent disruptions that tightly packed arrangements encourage are extremely detrimental to the flow of work. In another study, the same author measured the cost of interruptions² on daily office work. One might think that interruptions related to the task at hand are different from interruptions related to a completely different context. This turned out not to be the case: there is no such thing as a good interruption. “Our results showed that any interruption introduces a change in work pattern and is not related to context per se.”

Surprisingly, people took longer to complete tasks in the absence of any interruptions. But there was a caveat: “When people are constantly interrupted, they develop a mode of working faster (and writing less) to compensate for the time they know they will lose by being interrupted. Yet working faster with interruptions has its cost: people in the interrupted conditions experienced a higher workload, more stress, higher frustration, more time pressure, and effort. So interrupted work may be done faster, but at a price.”

How do we reconcile this? Being tightly packed with your coworkers does seem to offer significant advantages, but not without a price.

What is life like for PhDs in computer science who go into industry?

This is one of the most common questions I get during recruiting events. I have a PhD, and have worked in industry for last few years, so I am going to take a stab at answering.

First of all, I want to administer a dose of reality to the OP. Please don’t take offense at this, because I used to think exactly like the OP when I was a green newb. The entire set of questions has an undertone of “how will I be treated special in the workplace if I have a PhD?” You will not be treated special at all. When you go about your day-to-day job, you do not have a halo hanging over your head declaring that you did a PhD. You will earn the respect of your peers and bosses by delivering value to your team, not by possessing a degree. (Also see answer to job security question below.)

It is natural to feel special after a PhD. Academia implicitly drills it into you, because a PhD is academia’s entry ticket. You must be among the chosen ones if you have one. This privilege does not transfer to industry.

About the only concrete benefit of starting out with a PhD is that you will likely have a slightly higher starting salary, compared to those with a Masters.

There do exist venues in industry where research by itself is a specially delineated activity, and researchers are treated differently than regular engineers. They are the well known industry CS “labs”. A few years ago, the big four were: Microsoft Labs, IBM Labs, Sun Labs and HP Labs. They were shining examples of how research could be embedded in a industrial setting, advancing the state of the art while also having a positive impact on the company’s bottomline. But, they were clearly delineated from the rest of the company, and essentially served as an in-house CS department. One of the core activities of such labs was “technology transfer”, so that the cool things they came up with could be disseminated to the rest of the company. Today, the only two still going strong are IBM and Microsoft Labs, while the others have diminished.

What happened in the meanwhile was that Google started operating with a hybrid model of industrial research, where research was not a cleanly separated division or activity, with blended seamlessly into the regular operation of the company. There are people with the title of “researcher” and those with the title of “software engineer”, and a continuous gradation in between, with researchers expected to focus on more ambiguous and blue sky problems, and engineers expected to focus on more immediate implementation and design issues. This is a marked departure from the “labs” model. What it is essentially saying is that every problem that the company tackles is in some sense research worthy.

It was also around this time (early 2000’s) that the expectations of what PhDs should work on began to change. If you graduated with a PhD, the highest status job you could get was a tenure track university professorship. Indeed, most PhD programs implicitly gear graduates for that end. If you could not attain that, the next best thing to strive for was a researcher position in one of the aforementioned industrial labs. And, if you could not even land that, you would go looking for regular industry jobs with the unwashed masses.

The problem with this picture is that there are 10 PhD graduates for every tenure-track position. And, while I don’t have figures, the industrial labs don’t hire at a much faster rate either. And that left regular industry jobs as the only viable option for the vast majority of PhDs. My guess is that the recruitment engines of most major tech companies also wised up to this fact, and started actively pitching to not just bachelors and Masters graduates, but PhDs as well. The new crop of web companies, with Google at the forefront, also started projecting the image that the work being done there was new and cool and challenging and important enough to keep PhDs engaged. Also, the pay was pretty good. When you saw that year after year a significant fraction of the outgoing PhD graduates from your CS department were happily ensconced in regular industrial jobs, option number 3 began rapidly catching up with the other options.

• What is the day to day life like?

At the broadest level, you will spend about half of your time designing systems and features, and getting those designs discussed and reviewed, and the other half actually implementing those. (Also see the next answer.) As you become more senior, leadership/coordination activities also start becoming a significant part of the job.

You must be a great coder. That is a minimum prerequisite. During one interview, I asked a candidate (a recent PhD graduate) to give me some pseudo-code for the solution he had just described, and he went “oh well, if you must make me code…” That pretty much made me go “no hire.” What did you think the job involved?

• Do people usually work alone, or with others?

Again, the answer to this question does not depend on what degree you have. The simple truth is there is no job in the modern tech industry that involves working alone.

You will probably evenly split your time between collaborative activities such as design discussions and code review (both for your code, as well as others’ code), and solitary activities such as heads-down coding. The fraction of solitary time goes down as you become more senior.

• How much mentoring is there? Who can you ask for help if you need it?

Almost never an issue. Most places will actually pair you up with a mentor who is not your boss. This means you should feel comfortable discussing things with them that you wouldn’t with your boss. Also, everyone around you will also help you out, mentor or not.

Remember the crucial thing about asking for help in a way that will make people want to help you–show that you have put in some work and are stuck at some point. Don’t just go around asking questions because you are too lazy to dig.

• Are good jobs available in all locations? Are you usually required to move after grad school if you want to get a good job?

You are much more likely to land an interesting job at the coasts. West more than East.

• How many jobs are available and how many publications do you need to get them? How cutthroat is the competition for good jobs?

Most industrial positions will not care about your publications, unless you have none or very few, in which case the question will naturally arise of why you produced so little over your PhD.

Competition is high. You have to be prepared. You have to be specifically prepared for a tech/coding interview. Use your favorite search engine to get inundated with tips for how to prepare. This is a great starting point.

• Are you only qualified to work in the exact subfield you studied in grad school?

This is another big area of adjustment. Employees will usually try to match you to a position that relates to your research. But do not expect to spend your entire career are in the same field that you got your PhD in. (Why would you even want to do that? It would get boring pretty quickly.) Getting a PhD should have equipped you to tackle problems outside your area.

• How is job security?

There is no such thing as job security, no matter what degree you have. The only way to be somewhat secure is to have demonstrable competence. Note that I said not just competence, but demonstrable competence. If you are doing great things, but nobody knows about them, it doesn’t count.

All those skills you developed during your research career to promote your work and your papers, to get your point across, to put it in context, to explain why it is important–both in terms of written and spoken communication–will be crucial in developing your career. If you feel like you are not developing these skills during the course of your PhD, then you should really focus on them.

In some places, another way to get security is to be really good at internal political games. You do not want to be at a place like that. If you live by politics, you die by politics.

• Do you work 9 to 5, or do you take your work home with you?

This, again, is completely independent of what degree you have. How you structure your priorities and your work to get it done within sane hours, and while maintaining some sort of “work/life balance” (I hate that term, but that is a whole other story) is entirely up to you.

• Are you expected to publish papers?

If your title is “researcher”, then yes.

• How hungry are people for career advancement?

How hungry are you? It is an entirely personal choice.

• If a PhD goes into industry, does he usually do the same type of job that you could get with a Master’s? If not, how do they differ, and why are they better/worse?

Yes, you will usually do the same type of work.

At this point, you might be wondering “so are there any advantages to a PhD at all?” Yes, there are. You have to use the skills that you developed while doing a PhD to advance your career. “What might such skills be?”, you ask. The two most important are the ability to work with ambiguous and ill-specified problems, and communication skills. This is where PhDs should edge out others.

But, I’ve seen plenty of PhDs without these two crucial skills, and plenty of non-PhDs with. If all the problems, as well as the sketches of the solutions, that you worked on during your PhD were handed to you by your advisor, and you did not wrestle with any demons on your own, you will probably fall short here.

• Do most people stay in industry, or do they use their jobs as a jumping-off point to get back into academia?

Except in extremely rare circumstances, I’ve not seen people going back to academia from industry. There has, however, been a recent spate of people moving from academia to industry, even giving up tenure in the process!

• Do you ever regret getting a PhD?

No. Here’s why.

• Do most people have time to get married?

(See answer to 9-5 question above.)

• How are these impacted by the prestige of your graduate school?

Does not matter. The only thing that matters is how you perform on the interview.

To me, the crucial question is: what is the lure of work, specifically for engineers? I find it hard to believe that somebody pushed themselves to this level of personal sacrifice because their manager pushed them, or because they were coerced in some way. There has to be a deeper reason: they did it because they wanted to. Haven’t we all sometimes treated work as a refuge from home, rather than the other way round. Question is: why, in spite of the obvious costs? What is the lure of work?

I remember when I got my first computer, a PC AT, back in high school. I couldn’t get my hands off it for the next thirty six hours, at which point my body refused to co-operate. Afterwards, I spent hours every day on it, tinkering, learning, exploring… having fun, losing myself.

This is pretty much the story of every computer geek, right?

You grow up, you build deeper relationships, you get responsibilities, you have the entire world inundating you. It’s all very enriching and exciting. Yet, you find yourself always coming back to this inanimate object that is almost a shelter. Tinkering, exploring, learning… having fun, losing yourself.

The world of humans is messy, unpredictable, and–this is the part that infuriates you the most, the part you simply don’t get, the part that will forever make you an outsider–unreasonable. Humans are unreasonable. Your whole life, you’ve trusted one thing to save you. And saved you it has, again and again, without fail. That one thing is reason. Computers are a tool of reason, to be handled by people who obey the code of reason. People who are involved with computers, they’re reasonable, just like you. You all share a deep, unspoken bond. You’re all worshipers in the church of reason.

Welcome home.

The author delves into the history of programming as a profession, and also the question of whether it is a profession at all.

To me, one of the most fascinating bits was about how programming started out as a low status, mechanical and clerical job. The designers and creators of hardware were at the top of the hierarchy, followed by those who “designed” high-level solutions to problems, and then at the very bottom where the “coders” who would, literally, take these high-level plans and encode them into machine language. Only time and experience revealed that this was far from the lowly, clerical job that it was initially thought to be.

Another interesting thread was the continuous presence of Taylorism as a strong motivation for software engineering. From the very birth of the term at the 1968 NATO conference, the goal of software engineering was to make the construction of software just like the construction of physical goods, with a “software factory” being the ultimate goal. The strong intuitive and artisanal tendencies of the programming community were always at odds with this.

My only criticism is that it reads exactly like what it is–an academic dissertation turned into a book, and as such it can be a bit dry. This minor problem should not detract from an otherwise engaging journey.

The internet is full of productivity tips and techniques, more accurately known as productivity porn. It’s like McDonalds trying to sell you healthy food. You know that is what you should eat, but when you actually get there you end up eating the nutrition-free fast food.

And I plead guilty. I’ve tried them all. Every now and then I stare into the chaotic abyss of the stuff I need to keep on top of, and decide what will save me is a new system. Of course, it doesn’t.

I’ve learned a few things the hard way that are not often mentioned.

If you really deeply care about something, you will do it. You will do it without needing a list or a system or a reminder. No, your brain will not feel cluttered by the burden of having to remember it. If it happens to be a complex task with many steps, you’ll make a list, without thinking “oh look I’m doing GTD.”

So what do you need a system for? You need it for chores. The stuff you don’t want to do, but you need to. The stuff which is easy put off but will hurt in due time. Stuff like paying bills and calling customer service.

Do not confuse activity for progress. Sometimes blind activity can be a palliative, a false reassurance that we are moving forward. It takes courage to stop and ask if what is being done is actually having any impact–on our lives and our work and our happiness. We know the answer will often make us wince.

• Gotta have custom controls: If you use the standard UI controls, your app will look bland and boring and barely any different from the demo apps that come with the SDK. To get your app even a little bit past the threshold of noticeability, you really need to build your own custom control with its own unique look and feel. Every popular app does this. I can’t think of a single decent app that doesn’t.
• Design is everything: you have a tiny canvas (maybe not so tiny if you’re developing for a tablet, but still), but unlike the desktop, you have all of it. Also, unlike the desktop, there are very few rules to adhere to. Great design and user experience completely dominates the impact of the app.
• Your app has richer inputs: my desktop-bred keyboard-clacking geek brain immediately dismissed mobile devices as inferior because they didn’t have a keyboard. I can be stupid like that. The fact is, modern phones and tablets have richer inputs than desktops. Location. Gyro. Touch. Laptops and desktops are archaic and crippled by comparison.

When you ask if language or technology is worth it, the first thing you have to do is define what “worth” means to you. For the purposes of this post, I take that to be money, i.e. economic worth. In general, CS education does very little to teach us how economic factors play into technical decisions. The economics of programming has been a strong theme of this blog.

Let us start by asking the economic variables and commodities that should be considered when trying to answer this question. At the highest level, the economic variables that matter the most are:

• Material: this is the cost of hardware, real estate, electricity, bandwidth, and everything else that you explicitly need to buy in order to run your service or product. This is the easiest variable to account for, because you write checks and get stuff, and all the quantities and numbers involved are clear.
• People: this is the cost of hiring and maintaining the people necessary to build and maintain your service or product. This variable is more fuzzy than material, because there are many costs associated with hiring a person besides just their compensation, and these costs become known only in the aggregate. For example, a general rule of thumb is that the total cost to a company hiring a person is approximately double their salary.
• Time: the cost of time has two components–the monetary value of time spent by the people you have hired, and the monetary value of losing out in the market due to arriving too late or building the wrong product, i.e. opportunity cost. Due to the 2nd component, this is by far the most fuzzy and hard to account for variable.

The total cost of building and maintaining a product of service is the sum of these 3 costs. I’m going to assume that the intent of the original question was to decrease cost, while ignoring revenue and profit from the product.

I am by no means suggesting that these 3 are the only economic variables that should be considered, just that they form a crude model sufficient for the argument at hand. One could go on to list dozens of other variables, but most of them would fall into one of these 3 categories.

Now that we have set the stage, we can reframe the original question to “suppose we built a product using C or C++, and we know its cost to be X, if, in an alternate universe we built the exact same product using language L, would the cost be less than X?”

Note that this alternate universe does not have to be a parallel one, it could just be the universe starting at time=now, i.e. building the product over again with the new language.

An alternate way of framing the question would be: given the space of cost defined by these variables, in which regions of that space is language/technology X more economical than others?

Let me make this concrete by considering two common scenarios:

You are a startup: in this case, the cost of time dominates the other 2 variables. You are in a race to build something fast, release it, discover whether it works, and iterate. If you can use a technology or language that chews through material that lets you build and modify your product very quickly, you should choose that. This is a familiar story–and the reason why supple and expressive languages like Ruby and Python are so popular among startups.

You are a warehouse scale computing company: you run multiple large data centers, with gazillions of machines. You have a few products and services that are used by billions of users. At this scale, the cost of material starts becoming a significant component of your overall cost, and may even dominate the other two variables.

The difference between these two scenarios boils down to this: are you spending hardware on people, or people on hardware?

The major problem is that the crossover between the two is extremely fuzzy. It’s like navigating a ship across the Pacific with your instruments only telling you when you have crossed either Tropic. Chances are, one fine day the person who actually writes the checks will notice that, for example, they are spending nearly as much on material as on people, and will instruct the next line of engineering managers to crank the dial towards optimization.

The astute reader will notice that I still haven’t answered the question. The answer is… it depends.

The particular example that Lemire used is a batch job, where we would like to maximize throughput–process as many integers possible. Another important class of applications does not care so much about throughput, but about the latency of each individual request. VM randomness like GC pauses are a disadvantage for those.

VMs have their own memory overhead, which might not be significant when running on a few machines, but when multiplied by thousands of machines quickly becomes significant.

Is C/C++ worth it? Yes. For backend infrastructure that runs across entire datacenters, the truth is the economics simply do not work out with anything else. Of course, not everyone operates on that scale, and for them the answer will be different. How exactly C and C++ landed up in that position is an entirely different story. And the situation looks like it won’t change anytime soon, because while there has been an explosion of higher level languages in the last couple of decades, nothing has come close to dislodging C/C++ from the perch of systems/infrastructure supremacy.

One more thing: implicit in Lemire’s post is that the “low-levelness” of C/C++ is a substantial burden, and it would be a worthwhile tradeoff to use a higher-level level, more expressive language. In my anecdotal experience, the low-level nature of C/C++ has not been an issue with the right programmers. I’ve seen programmers around me who are productive in spades with C++, notwithstanding all the blogosphere’s chatter about how much it sucks to work with C++. I think the reason is that the bottleneck in system construction is often elsewhere. Again, that’s a whole other story.

So the answer is, yes, C/C++ worth it when it is.

This is important, because all projects start with a big blob of vagueness. Folks who create the most concreteness out of that are the ones who have the biggest impact. I’m learning how to approach this, but I still feel like a non-swimmer in the shallow end with lifesavers. I’ve learned a little, by personal experience, and by observation.

The design space is huge and completely unconstrained at this point. There are too many unknown unknowns. How does one proceed?

First of all, formulate principles that will guide the solution. This is not design yet. These are high-level ideas that put constraints on the design space. For example, GMail chose the principle that search should be the primary email navigation mechanism. Think long and hard about these, because these principles will be baked into your code and your system in a way that can’t be “refactored” away later. This is where wizards (i.e. very senior battle-hardened engineers) can help. Often, choosing these is a matter of taste and design and conceptual integrity. It might sound wishy-washy but it’s hard. Document these well. This is your manifesto.

Once you’ve got a spine, you’ve got to add a few more bones.

The next step is to find one concrete idea, and build on it. This usually takes the form of a prototype, or a walking skeleton. It might have huge swaths of functionality missing. It might have serious shortcomings. It might be ugly. It might be slow. But that’s OK. Think of this as the spec of sand around which a pearl is formed. What is important is that it be concrete, and growable.

It needs to be concrete and tangible. Something you can show and demo and people can play with and talk about and criticize get excited about. People rarely get excited about design documents. They frequently get excited about working prototypes, even when they come with massive this-will-break-on-you disclaimers.

The other crucial thing is that it should be growable. Your prototype should not be a dead end. You should be able to put work into it and make it incrementally better. Adding something new or changing it should not require you to build a new one from scratch. In other words, it should support iteration. There may come a point when you have to throw one (or more away), but ideally that happens only when the prototype has brought you to a major insight that invalidates some of your basic assumptions. Even then, it would have served its purpose. And even then, you need something that will get you to that point quickly.

Which brings me to my most important point: iteration. Speed of iteration almost always beats quality of iteration. In other words, focus on learning and fixing, continuously and quickly. Don’t sweat too much about getting it right. Chances are, your idea of right won’t be the right one. Chances are that most other peoples’ idea of right won’t be either. Only by having something real to criticize and learn from will everyone figure out what right is.

Build, fail, fix–and repeat.

What was missing was an appreciation of how, and why, particular data structures are used in the real world, embedded in real systems.

So I’ve been wondering, while teaching a DS course, what would be a good way to show the students non-toy examples of how they’re used?

I think the Linux kernel is an excellent example.

Let’s look at how the kernel puts some data structures through the paces.

Red-black trees

From the kernel docs:

There are a number of red-black trees in use in the kernel. The deadline and CFQ IO schedulers employ rbtrees to track requests; the packet CD/DVD driver does the same. The high-resolution timer code uses an rbtree to organize outstanding timer requests. The ext3 filesystem tracks directory entries in a red-black tree. Virtual memory areas (VMAs) are tracked with red-black trees, as are epoll file descriptors, cryptographic keys, and network packets in the “hierarchical token bucket” scheduler.

The kernel implementation is different from most naive implementations, and that teaches you something about performance and cache locality:

The Linux rbtree implementation is optimized for speed, and thus has one less layer of indirection (and better cache locality) than more traditional tree implementations. Instead of using pointers to separate rb_node and data structures, each instance of struct rb_node is embedded in the data structure it organizes.

Radix trees

Again, the kernel imlementation is different from the standard one:

A Linux radix tree is a mechanism by which a (pointer) value can be associated with a (long) integer key. It is reasonably efficient in terms of storage, and is quite quick on lookups. Additionally, radix trees in the Linux kernel have some features driven by kernel-specific needs, including the ability to associate tags with specific entries.

The most widespread use of radix trees, however, is in the memory management code. The address_space structure used to keep track of backing store contains a radix tree which tracks in-core pages tied to that mapping. Among other things, this tree allows the memory management code to quickly find pages which are dirty or under writeback.

Possible Assignments and Questions

• The kernel doesn’t have any “fancy” data structures. For example, there are no skip lists? Why do you think that is? (E.g. see these attempts)
• Swap out a data structure with another one (AVL trees instead of red-black?) and see if can improve performance, or simplify the code.

Such exercises place the data structure in the larger context of a whole application and real workloads, which is much more interesting than just implementing a standalone data structure and exercising it.

I use Readability’s “Send to Kindle” bookmarklet.

And for my RSS feeds, I use Calibre, which can take a list of your subscriptions, download them, package them up nicely as an ebook, and send them to your Kindle.

I think most managers would ideally prefer their world to be like a turn-based game. One where the other side makes a move, and then nothing else happens while you observe the state of the world, use all your faculties, then make your “move”. Repeat ad infinitum.

The worst managers follow this mental model, possibly not even realizing that it has no reflection in reality. Mediocre managers realize that reality is different from this model, but try to act as if they can play out a coarse approximation. And great managers (or at least, those that strive to be great) completely reject it.

Growing up, we used a lot of paper. A lot of color pencils and crayons. Our teachers put us through endless drills in cursive handwriting. A neat, legible, and beautiful hand was something to be strived for, something that was prized, and rewarded and shown off.

We had long afternoons to ourselves. We had a loyal band of neighborhood friends. We would have four hour long play sessions. Sometimes, we would listen to entire albums from beginning to end–while doing nothing else. Do you even remember the last time you just listened to music, without it being a soundtrack to some other activity you were doing?

Sometimes, we ache to go back to that time. That time seemed simpler and purer. So much so that we are willing to mutilate memories from our immediate past with sepia and Polaroid filters. Nostalgia is painful, but it is also sweet and powerful.

But here is the thing: nostalgia is a trap. It is not that those times were simpler and purer. We were simpler and purer.

Nostalgia is easy to fall into. And the older you get, the easier it gets. The universe of things you can look back on only increases with time. And it seems so much more pleasant than looking forward, where you only see hopes and dreams and fears and probabilities. It takes conscious effort to not go down that slope, to instead look to the future, and actually create it. And it takes even more effort, and more courage, to objectively compare the past to the present, and face the fact that, yes, indeed, most things are better, and are more likely than not to continue getting better.

Over the last year, I have found myself writing by hand again. Sometimes, it is page after page of straight prose. Sometimes it is phrases and bullet points and underlines and bubbles. Sometimes it is just random senseless doodling. And the reason I have come back to that archaic activity is my LiveScribe pen. I no longer have to worry about losing all that. Something that is naturally analog and free-form is seamlessly brought into the digital world.

We seem to be enveloped by the literature of despair and frustration. Complaints and pessimism always seem to be more profound and erudite when placed next to cheerful optimism. Reject that.

Look forward. Make the future.

A synchronous working style is one where some of all of the team has to regularly stop what they’re doing, meet and discuss and agree on the next steps together, and then get back to work implementing those steps. These synchronous “join points” are frequent, unavoidable, and built into the structure of the work.

An asynchronous working style is one where the entire team rarely, if ever, gets together for big agreements and discussions, each individual team member more or less works on their own and at their own pace, with collaboration and agreement being handled by asynchronous mechanisms such as e-mail and shared online documents.

With these definitions, geographic location and working style are largely independent. It is possible, and even common, to have a team that sits together in an open cubicle arrangement and yet works mostly asynchronously. Conversely, it is also possible to have team members geographically strewn, and yet having to synchronize frequently.

Working asynchronously certainly enables the team to be geographically distributed, or telecommute, or maintain flexible hours.

The asynchronous style centers around shared artifacts, with each team member gradually accreting layers and complexity onto those artifacts. The most common example is commits to a source code repository. Other examples: code reviews (which just collapse down to e-mail), shared online documents with multiple editors and commenters. Team members usually have multiple related threads of work going on simultaneously, so that if one is blocked on input from someone else (for example, if you are waiting for someone to review a document, or comment on your code), another thread can be picked up to make forward progress.

Of course, most of us will recognize our team’s style as a hybrid of synchronous and asynchronous. But the trend towards the asynchronous style is clear and strong.

Most software companies work asynchronously, even if team members sit next to each other. The common picture is one of an open cubicle farm, with everyone wearing large headphones. It is pretty common to use instant messaging with the person two desks down, rather than actually talking to them, especially if they have the big fat “do not disturb” sign that is the headphone.

One sign of things being asynchronous by default is that I find it archaic, even rude, when someone insists on calling me on the phone. It is an unnecessary imposition on my time, and breaks my flow. Why can’t you just e-mail me?

Update, June 26, 2012: I recently came across another interesting paper in this vein: If Money Doesn’t Make You Happy, Consider Time. It talks about how we should spend our time to increase our happiness.

And so it was a pleasant surprise to read this paper that makes the case for studying and understanding the social factors that affect the adoption of programming languages and programming features.

(Funny story: I found this paper at the printer in the office, glanced at the title and ran back to my desk to Google it.)

Don’t expect any grand answers. The paper raises many more questions than it answers. It’s special because it makes the case that social factors are critical when considering adoption of languages and their features, and proposes a research agenda for investigating them. It is also a thorough survey of the scattered remarks and papers by language designers where they reflect on the adoption (or lack thereof) of their creations.

I would’ve also loved to see some discussion of the economical and organizational factors that affect PL adoption. For example, the criteria that companies use when choosing languages are very different than those used by individual programmers. Using Lisp (or your favorite why-the-hell-won’t-they-let-me-use-that language) might make you super-productive (or so you think) but will it scale to making the entire team productive as well? Early hires probably have much more room to choose. The large majority of working programmers enter a situation where the language is already chosen and they have to work with it.

Another factor they miss is the context around a language. Languages are seldom chosen in isolation. Usually, much more thought goes into choosing a platform, and the language comes with it. Think about the Web and Javascript, Unix and C, and so on.

But overall, a great read and an agenda that was a long time coming.

The instructors are going to enjoy it more.

Why? Don’t they prefer the live interaction of a classroom, the feel of chalk on board?

No. If you think about it, classroom teaching, especially at the university level, has been steadily declining in quality. The professor comes in and rattles off a stock deck of slides off their laptop. The students all have their laptops open, web surfing.

It’s unenjoyable for the students, because they’re just sitting there, postponing the learning (or cramming) until finals week or the the day before the assignment is due. It’s painful for the professor, because they have to look out into a sea of disinterested people. Once in a blue moon, a sharp, interested student will really engage with the teacher and the material and make it worthwhile.

The interested minority is drowned by the passive majority.

But look what happens when the material is online. The teacher only hears and engages with the interested students, and there are a lot more of them. That is so much more motivating. The forums, the email, the blogs, the tweets, the questions, the corrections—they’re all from the interested minority. But it feels like they’re the only ones there.

Suddenly, the interested minority is is in control.

The category representative of the genre is Ronald Azuma’s timeless classic “So Long, and Thanks for the PhD.” Much of what I wanted to say has already been said there, and although the piece is more than a decade old, it holds up remarkably well. What I want to do here is provide some personal perspective.

My credentials are that I’ve actually done a PhD in computer science, and since then spent as much time out in the real world as in grad school, so I’ve seen both sides of the fence.

Do you really want to get a PhD?

Before climbing the ladder, make sure it’s up against the right wall. Should you get a PhD?

The answer to that depends on two questions? Do you need one? Do you want one?

You need a PhD if you want to go into academia. If you find the process of getting a PhD unenjoyable, that is a strong sign that you are not cut out to be in academia. If a PhD makes you hunger for more, then you are.

The numbers are stacked against you though. PhD graduates in computer science outnumbered tenure-track professor positions in CS by a factor of 10. The next best option for those who want to be in academia is to become a postdoc,which is why postdoc numbers surged, but let’s face it, that is a crappy second option.

And let’s not even talk about the financial oppurtunity cost. It is nearly 100% certain that someone who started in industry on the day you joined grad school will overtake you in salary by the time you graduate and join the “real world.”

There used to be certain positions in industry, especially in R&D, that absolutely needed people with PhDs. That has been steadily eroding over the years, and while a PhD is certainly a plus, its necessity has been watered down significantly. Even if you do not formally have a PhD but can make an impact and prove yourself in an industrial research setting, you can perform the exact same job as someone with a PhD.

Why am I being so discouraging here? Because I have seen my share of people get into grad school for the wrong reasons and go on to waste their time at best or get stuck in an unhappy path for life at worst. If you realize this is not for you, cut out after your Masters.

What it boils down to is that this is one of the most intense questions of self-knowledge you will ever face. The answer is simple: you should do a PhD if you really want to. Look into yourself to figure out if you really want to. Everything I said above was to list out things that should not factor into your decision.

This is what I like to call a clamped decision. If you are not fully convinced that the answer is “yes”, then the answer is “no.”

So why should you do one? I’ll get back to that after considering the more practical question of choosing where to go.

Choosing where to go

Assuming you are brilliant enough to have scored multiple admission offers, choosing where to go is a big decision.

What should your criteria be? If you are like most fourth-year college students, you will only look at one thing–the rankings of the schools. This is a big mistake. (With one important exception. See below.) The choice is much more nuanced than that. There are actually three big factors at play here, and you have to decide which ones you value more, and what balance to strike among them. They are:

• Your happiness and quality of life during the 5 to 6 years you will be in graduate school
• The quality of the learning and research at the school
• Your future prospects after graduation

Your prospects after graduation have have nothing to do with your school if you are going into industry. Industry (at least, the tech/software industry) does not care about your alma mater. They care about whether you can program. They do not care about your credentials. They care about whether you do well at their interview.

There is an important exception. If you want to go into academia after your PhD, then school ranking matters, because whether they admit it or not, in practice a school will only hire new professors who have graduated from schools close to their own rank. This works the other way round too. Candidates will only want to join schools with ranks close to the one where they graduated from.

Which leaves us with the first two criteria.

The one choice that affects both of these is who your PhD advisor is. That should make up 80% of how you choose where to go. Repeat after me: PhD advisor. PhD advisor.

Are they someone that will nurture you, and give you room to grow and explore, while not breathing down your neck, and having the wisdom to gently nudge you back on track when you will inevitably be lost in the dark jungle of the trough of your PhD? Are they someone with a track record of successfully funding their research, so that you and your project are not suddenly left high and dry without money in the middle of your PhD?

How is one to find out these things? You can look at their webpage, but take everything there with a grain of salt, just as you would someone’s resume. A better way is to email their current grad students. You would be amazed at how helpful they might be. A large group of grad students is also a good sign of being funded.

I have seen first-hand what a stressed out and crappy life some of my fellow grad students had due to their advisors insisting that they put in a certain number of hours at the lab everyday, or not letting them graduate even though they had done more than enough work and published more than enough papers to justify getting a PhD already.

Graduate school is a long haul, and almost your entire happiness and quality of life depends on your PhD advisor. I cannot stress this enough. This is the only choice you should be stressing out about. All other variables are in the noise.

Coming to secondary factors, the next most important one is location. I wish someone had impressed this on me. I chose California, but that was luck, not wisdom.

You will have a much richer and more stimulating life if you are in a densely populated area as opposed to a small town with a ninety minute drive to the nearest big city. If you are in CS and want to go into industry, try to be on the coasts, preferably the West one. If you come from a place that does not have long snowy winters, beware of places that do. The “oh look the snow is so beautiful!” feeling goes away very quickly after a few months of shoveling snow.

Why do a PhD?

Let’s come back to the question of why you would want to do a PhD.

Like I said above, this is a question of self knowledge. The circular answer is that you should do it if you are the type of person who would enjoy it.

Let’s take another excursion and ask: what is the value of a PhD? Which begs the question: value to whom?

The value to employers is that it indicates deep specialty in a narrow field. Or so one might think. Does that mean that when you dissertation topic is no longer fashionable and becomes obsolete you have no value? No. So it must be more than that.

Another value to employers is the indirect signal a PhD sends. It means you took an a substantial intellectual task and stuck with it for a long time to achieve some sort of closure. That is valuable. But even more valuable than that is the fact that you faced an ambiguous ill-specified problem and charted it through an uncertain path. That is the central value of a PhD: to be able to comfortably work with ambiguity and uncertainty. Note that this implies the serial-learner “learning how to learn” capability usually ascribed to PhDs.

I’m certainly not claiming that doing a PhD is the only way to get the capability of comfortably working with ambiguity and uncertainty. There are certainly other activities and experiences in life that can equip you for that. But a PhD is structured to do that.

But I’ve only spoken of value to someone else. What about the value to you, as an individual? That would include all I’ve said above, but what else?

Before I come to that, I want to take a slight detour to talk about boredom, solitude and entertainment.

William Deresiewicz on solitude and boredom:

Boredom is not a necessary consequence of having nothing to do, it is only the negative experience of that state. Television, by obviating the need to learn how to make use of one’s lack of occupation, precludes one from ever discovering how to enjoy it. In fact, it renders that condition fearsome, its prospect intolerable. You are terrified of being bored — so you turn on the television.

I speak from experience. I grew up in the 60s and 70s, the age of television. I was trained to be bored; boredom was cultivated within me like a precious crop. (It has been said that consumer society wants to condition us to feel bored, since boredom creates a market for stimulation.) It took me years to discover — and my nervous system will never fully adjust to this idea; I still have to fight against boredom, am permanently damaged in this respect — that having nothing to do doesn’t have to be a bad thing. The alternative to boredom is what Whitman called idleness: a passive receptivity to the world.

So it is with the current generation’s experience of being alone. That is precisely the recognition implicit in the idea of solitude, which is to loneliness what idleness is to boredom. Loneliness is not the absence of company, it is grief over that absence. The lost sheep is lonely; the shepherd is not lonely. But the Internet is as powerful a machine for the production of loneliness as television is for the manufacture of boredom. If six hours of television a day creates the aptitude for boredom, the inability to sit still, a hundred text messages a day creates the aptitude for loneliness, the inability to be by yourself.

Hal Abelson, co-author of SICP, said of entertainment and education (and this is entirely from my memory of a talk by him I attended six years ago, and is paraphrased, so is almost certainly warped, which is why I’m not even going to put it in quotes): We see all around us that the marginal cost of goods and services eventually approaches zero, but one thing we all want and pay for is entertainment. And that’s the value of education. Ultimately, the more educated and creative you are, the easier it will be for you to entertain yourself and never be bored.

And that finally brings me full circle. The value of a PhD to you is that you will never be bored, and that you will be comfortable with solitude and cherish it. This is partly why I keep referring to universities (especially grad school) as the closest things to modern-day secular monasteries.

That said, I want to point out one glaring omission. I realized its importance only after working in the real world for a few years, and being schooled in how inadequately prepared my CS education had left me in this arena.

That topic is operations. It encompasses all the activities required to keep a computing service deployed, performant and available. Chances are, if you work in the industry today, even if your title is something like “programmer” or “software engineer”, operations will be some component of your job. The fraction of time you spend on operations might fluctuate between 0% and 100%, but it will certainly exist.

In broad strokes, operations includes the following:

• Production design: okay, your shiny new system is code complete. How exactly will it be deployed? How many machines? What kinds of machines? What resources does it need? What are the failure modes of your system? What happens if x% of your machines go down? What happens if all your machines go down? Under what failure circumstances can the service continue to limp along, as opposed to being completely unavailable? The answers to all these questions makes up the production design of your system.
• Deployment: how would you automate the process of pushing new code? Of taking the entire system up and down? Taking subcomponents of the system up and down? How will you do a rolling restart of all your machines?
• Monitoring: how will you get visibility into the performance and health of your system? This includes inserting hooks to measure key characteristics (qps, latency etc.) as well as collecting and visualizing them automatically, in a way that will make sense to a bleary-eyed person who has been woken up by a page at 3 AM.
• Alerting: okay, you’ve done a great job of monitoring the health and performance of key system metrics. Now how will you notify yourself when something goes wrong? This includes figuring out the exact thresholds of the metrics after which the system is deemed unhealthy. This really needs to be foolproof. You don’t want your system to be down, and not even get paged about it.

In short, being responsible for the uptime of your own code gives you a whole new perspective when you write it. You start thinking about what could happen in there that might page you at 3 AM. You also realize how writing code is but one step in the never-ending cycle of deploying, tweaking, and writing again.

Where might this fit into a computer science curriculum? One option would be to make it an additional component of existing programming exercises, by having students deploy their programs on test systems and simulating failures, and then seeing how well their monitoring and alerting holds up. An interesting twist might be to have students be responsible for the upkeep of not their own but other students’ programs.

Also see: the CS assignment I wish I’d had.

If, in the above paragraph, you substitute “writing” for “oral argument”, and “coding” for “writing”, it will sum up the current debate on whether you should learn how to code.

But the whole debate on learning to code is missing the larger point of the thought processes and mentality it takes to code. Like Dijkstra said, computer science is as much about computers as astronomy is about telescopes.

After all, why should the mayor learn how to code? Is he now going to start writing apps?

The current debate also entirely misses the question of gradation. Take mathematics for example. It is pretty important for the population at large to understand arithmetic, and possibly also some algebra. That’s why everyone learns those topics in school. But is it equally important for the entire populace to be well-versed in the intricacies of group theory and vector spaces? Probably not.

Instead of talking about coding, we should be talking about computational literacy. Being fluent in a programming language is just one aspect of that, and a pretty advanced one. There are other topics and ways of thinking that need to come first in the curriculum of computational literacy. And perhaps, some of those ideas and ways of thinking that come before coding in this ladder are sufficient for the general population.

Note that computational literacy has absolutely nothing to do with being able to fluently use computers. Just like being a good driver doesn’t mean you have a good grasp of mechanical engineering.

Jeanette Wing refers to this as computational thinking, and makes the case for it being a useful interdisciplinary mode of thought¹.

To build the case for broadly teaching computational thinking, we need to look at the major transitions in modes of human thought through the history of civilization.

The earliest known mode was perhaps what we now know of as the Greek school, when knowledge was held in brains and created and transferred through an intricate and time-consuming culture rooted in oral debate and discussion. This was spoken thought.

The next big transition was to written thought. This was the first major leap in the scalability of thoughts. Their total number was not limited to the capacity of human brains. They could be transferred without the presence of the original thinker.

The next leap along the same lines was printing–which made thoughts orders of magnitude easier to reproduce and spread. This is what truly cemented the supremacy of written thought.

And now we’re seeing the beginnings of the next major mode of thought–executable thought.

Executable thought arises when written things can be automatically consumed and manipulated. This breathes life into what previously was just an inert string of symbols. The most obvious example of this is a computer compiling and executing something written in a traditional programming language. This is also the thing that most often hooks people into becoming programmers–the idea of converting pure thought into action.

But it does not have to stop there. The form of the idea does not have to be “code” in the traditional sense. Take, for example, something as innocuous as a hyperlink. If you were to think about it with the old written thought frame of mind, it would be just a citation. But if you were to think about it computationally, and with a rudimentary understanding of page rank, that one link would become the trigger for a sequence of computational actions that have implications for the authority, trustworthiness and findability of both sides.

Another example: biologists have been better able to understand a number of processes when wearing their computational, rather than the life sciences, hats².

This idea of thought being executable already permeates most of the commercial and physical infrastructure of the world around us, and very soon, someone who cannot think this way will feel like an alien in it. To put it another way, computational thought will become a necessary prerequisite to function in a computational world.

At its core, an Easter egg is a programmer’s way to show off his artistry in an overt and obvious way.

The artifact that the programmer creates–a word processor, a spreadsheet, a search engine–is meant to be utilitarian. The code he crafts towards that end might possess a beauty seen only by his eyes, but to the rest of the world it is only a means towards an end, and that end is to serve a specific, well-defined function. The world judges his beautifully crafted code purely by its utility. To them, he is an anonymous constructor.

This is where the Easter egg comes in.

It breaks through the calculus of utility and presents something purely frivolous. It serves no purpose, other than to declare “I make cool things! I made this!”

It is a secret, revealed only to a select few. Out of the hordes of users a program might have, it calls out to the select few who have the curiosity and inclination to peel back the covers. “You want to see something cool? Here it is!”, it says to them.

But the fundamental urge an Easter egg satisfies is to declare to the world–there is art over here! You might not see it everyday, but if you pull this lever just so you will be treated to something awesome, and it will change the way you see this functional piece of software. You will get a glimpse into the artistry and the humanity of those who made it. This is their signature.

1. Shock & Denial: But… but… how can this happen? I have unit-tests! And they even pass!! Besides, look at the code, it’s obvously correct. Really, this shouldn’t be happening. Can you even reproduce this?

2. Pain & Guilt: OMG. The other guy just gave me a reliable reproduction. My code sucks! I’m a worthless moron! How many other bugs are lurking in the code I shipped? I should just quit programming and go into sales.

3. Anger & Bargaining: To the bug reporters: are you sure you’re calling my code with the right parameters? Maybe there’s a bug in your code. I think my module works as per spec. Maybe you should change the way you call it. Frickin’ morons!

4. “Depression”, Reflection, Loneliness- Life sucks. I feel like puking everytime I see a chirpy colleague checking in code by the shovel-full. I don’t even feel like going into the office anymore.

5. The Upward Turn: Ohhhh… I see what might be wrong. Hack, hack. Oh! I see where the bug is!

6. Reconstruction & Working Through: Hack, hack. It’s fixed! It’s fixed! I’m a rockstar coder again! I’m getting back to shipping regular code again.

7. Acceptance & Hope: I know code will always have bugs, no matter what. Do your best. Test, test, test. And never feel invincible.

Goals

Before I even get into the details, I must describe the overall goals that I want and need my computing set up to achieve.

Simplicity

As time goes on, time becomes precious. I spent years in college and grad school maintaining my own Linux boxes and servers, tweaking their every detail, compiling my own custom kernel, installing all the latest packages, and happily staying up all night to do so. Believe me, it gets old. I guess it was a good learning experience, but that is certainly not something I want to include in my regular set up. I want it to be as hands-off as possible. If there are services that meet my needs and handle things for me, I want to use them. I want to minimize the number of moving parts, both in hardware and software. I want things to work out-of-the-box. I want to spend the minimum amount of time possible tweaking and maintaining my setup.

Functionality

As functionality goes, my needs are quite pedestrian. Of course, I need my set up to be programmer-friendly, seeing as that is my day job. I don’t do anything that might be considered heavy duty, like editing large videos. I do deal with the constant stream of images and videos that my setup ingests from digital cameras and phones, but that is a mainstream requirement. By and large, the vast majority of my digital activity centers around consuming, creating and massaging text, be it structured in the form of programs, or unstructured in the form of prose.

Preservation

After you go through a few machines, you realize that they are nothing but shells, and in the grander scheme of things, completely inconsequential. What does matter above everything else is your data. In your digital life, you don’t just do things, you make them, and save them. Everything else is moot if all this data cannot be safely and verifiably be backed up for the long term. Think about what your life expectancy is. Now think about something recent from your digital trail that really matters to you. It might be vacation pictures, a journal entry, a blog post, anything. Would you like to have that digital item available for your perusal when you are 80 years old? Now try to remember the oldest digital artifact that you can still retrieve, and more importantly, decode and view. Can you? How old is it? That should give you some idea of how hard and full of unknowns the problem of long-term preservation is.

Your body

I’ve talked about taking measures to proactively avoid RSI previously on this blog. Long story short, if a significant fraction of your working life is spent working with computers, and you do not yet have even the mildest RSI, you should consider yourself extremely lucky, but not immune. Act like you do have RSI, and change your set up right now to avoid it. It will happen sooner or later, and you might as well tackle it before it actually happens, so that you have a good chance of completely avoiding it.

After years of experimentation, I have found that the following works for me:

• Absolutely no mice. As early as college, my right wrist was completely ruined by excessive mouse use. I moved to using the mouse only with my left hand, and also doing as much as possible with the keyboard, but I could feel the pain creeping into my left hand as well. I tried trackballs, vertical mice, ergonomic mice, everything you can think of. Eventually, I was saved by the magic trackpad.
• Use a split keyboard. Using a split keyboard, the kind that has an inverted V shaped space down the middle, lets me type without bending my wrists at an unnatural angle. I use the GoldTouch keyboard.
• Use dictation software. Of course, this does not work for programming and fine text surgery, but for tasks where I just need to transfer a large number of words from my brain into a computer, this works wonderfully and keeps my hands completely out of the loop. It still feels strange sometimes, because my entire life up to now the circuit that moves words from my head to the screen has been silent, and it takes some adjusting to speaking them out loud. But maybe that is just a ghost in my head. I use DragonDictate, the Mac version, and have been a happy user for almost 2 years.

This is what my keyboard/trackpad setup looks like:

Hardware

The piece of hardware that I physically interact with the vast majority of the time is the 13 inch MacBook air, the model from mid 2011. It has 4 GB of RAM, and a snappy 128 GB flash drive. Once you taste the sweet speed of a flash drive, you will never go back, and the smaller capacity compared to spinning rust drives will feel like an inconsequential sacrifice. It is by far the sweetest, most lickable piece of hardware I have ever had.

Before my current Mac phase, I had used Windows for several years. I honestly don’t see what the fuss is about–I’ve moved back and forth between the 2 worlds relatively easily. Maybe that’s because all the software that I regularly use runs equally well on both sides. The major advantage of the Mac side of things is that they tend not to suffer from the bit rot that makes a Windows machine completely unusable after about 2 years.

I’m 90% ready to make the jump to pure netbooks, with nothing but a browser and a net connection. About the only thing holding me back at this point is good dictation software.

For serious, heads down work, the MacBook is connected to a large 30 inch monitor, and an external keyboard and trackpad. I use the Apple Magic trackpad, and the Goldtouch split keyboard.

Most of the real work on the job happens on a humongous, powerful machine, that sits under my desk, fed only by a power cable and ethernet. This, of course, runs Linux. I just ssh and X11 into it from the laptop wherever I am and go about my work.

So those are the 2 major pieces of hardware I regularly use–the large machine that stays put, and the feathery laptop that goes around with me. It’s pretty simple. The laptop is the pretty front-end, and the desktop serves as the heavy backend.

I like this setup because I get the UI sweetness of MacOS X, while being able to use the dev setup on the Linux box.

Other miscellaneous pieces:

• the Logitech H555 headset for dictation, and video conferencing, and occasionally listening to music.
• Bose QC15 noise canceling headphones for the times when I really need to block out the outside world. They definitely go on flights with me. They are kind of expensive for headphones, but they really do make a huge difference. In a regular office environment, it’s as if–whooosh–the whole world just fell away. And on planes, the loud drone becomes a whimpering hiss. This is the one gadget that still amazes me every time I use it.
• An external USB drive, for backups. I use the Western Digital Passport 1 TB.

Software

Editors

This is where the religious wars usually start. And rightfully so. If, like me, your digital life mostly revolves around text, then you need to have a religious fervor for your text editor. When I meet a geek who does not feel strongly about their choice of text editor (or worse, uses notepad or gedit), all my “fake geek” alarms go off. As far as software goes, this is the centerpiece of the entire set up.

You know what’s coming–Emacs and vi–those are the only serious contenders. I have some measure of objectivity here, because I have used both for a number of years, and feel comfortable with either. In the end, I fell towards the Emacs camp. What finally tipped me over was the customizability. People keep talking about the look and the key bindings and how many keystrokes it takes to do something. But all that is not important. What is important is whether you can make this editor your own. In the final analysis, Emacs is not really a text editor, but a platform on which to build your workflows. It just happens to be really good at text.

But–I’m not advocating a particular editor here. It really doesn’t matter which one you end up with. What does matter though is that you are a power user of it. It should feel like second nature to you. You should have no trouble making it do your bidding. It should make you feel powerful and light and frictionless. When people look over your shoulder at you going about your work with it, it should look like an incomprehensible buzz to them.

Also, don’t be a text editor mono maniac. I love Emacs, sure, but I confess to dropping down to other tools when they are better suited to the task. For example, for dictation I usually use TextEdit, because MacSpeech Dictate can’t really work with Emacs’ text control. When I’m writing Markdown, I often use Byword. Also, for tracking my tasks, I use Omnifocus. I know, that can be subsumed by Org-mode in Emacs. Org-mode can subsume everything. Doesn’t mean it should.

Web browsing

Actually, I lied about the text editor being the centerpiece of my setup. I ran a time tracking utility, and found that even on days and weeks when I’m mostly heads down programming, the application I spend the most time in is my browser. So it’s pretty important to get this one right, and to tweak it exactly the way you want it.

I have settled on using Chrome. The one indispensable plug-in I use is Gleebox, because it gives me keyboard control over common actions such as selecting and clicking links, and switching tabs.

Terminal/Shell

I use bash. I don’t use a terminal application. I just run all my shells in Emacs. That way, they are simply text buffers, like everything else. Once you do this, you will never understand why people use terminal programs where the only editable line is for the command you are entering.

Digital photographs

I like to meticulously name the directories in which I save digital photographs. Usually, I also give each individual picture a short, descriptive filename.

For browsing them, as well as making small edits, I use Picasa. It is overall a great application. But what really attracts me to it is that, unlike most other photo managing apps, Picasa works with your directory hierarchy, and never messes with it. It is a read only application, unless you explicitly modify something yourself.

This is a useful litmus test for almost every application I use. It should work with my directory layout, and provide a view over that. It should absolutely never create its own directory layout with jumbled names that are opaque to me, and have me depend on that one specific app to view my data.

Backups and preservation

I keep all my digital artifacts in a version controlled repository. There is a “master copy” that I commit new items to, and check out from. This holds everything, including binary blobs such as digital photographs. The repository is the Golden copy. It is backed up many different ways. Rsync helps a lot with that.

This means that what is on my laptop is just a cache, a checked out (small) subset of the entire repository.

I know what you’re thinking–what version control system do I use for this? I use CVS. There, I said it. Back when I started my repository in 1999, CVS was cool and the state of the art. And now, I have so much precious data in my repository that I’d rather not migrate it to something else. CVS also has the nice property that it’s repository (the “,v” files) are human readable.

Common flows

Reading

Most of my reading comes through my RSS subscriptions, and then there are the occasional longform pieces I stumble across. But here is my twist: I listen to almost all of my reading. I dump it out to text, and then use Mac OS X’s speech synthesizer to “speak” them out into sound files, which I then listen to on my iPod. Why? It started because I wanted something interesting to listen to during my commute. But then I realized what a better experience listening to something is when compared to reading it on the screen. It’s easier on the eyes, of course, but also free of all the distractions that a screen brings. It also leaves you free to do other tasks, like driving, that don’t divert from the cognition required to comprehend the piece.

Wish list

The number one thing on my wish list is a high-resolution large (13 inches or more) display. I want something that comes close to the retina resolution levels of the iPad. This point was driven home to me when a full resolution screenshot of the new iPad did not fit on my 30 inch monitor.

Next wish: more storage. You can never have enough storage. Right now, I’m compelled to only carry a small fraction of my repository on my laptop. It would be great to have the whole thing, preferably at flash-like speeds. But I would even settle for a hybrid approach where the OS and apps live on a fast flash disk, and my deep archives are on slower spinning rust disks.

It looks like the bottleneck for advancing science is now not so much the presence of brilliant brains, but the presence of an advocate with charm and clout and connections, who can coax funding out of reluctant public bodies, and then charm them over many years while the project continues.

But even when such a person is at the helm, funding is always a struggle. Weinberg is pessimistic about public funding of big science in general:

What really motivates elementary particle physicists is a sense of how the world is ordered—it is, they believe, a world governed by simple universal principles that we are capable of discovering. But not everyone feels the importance of this. During the debate over the SSC, I was on the Larry King radio show with a congressman who opposed it. He said that he wasn’t against spending on science, but that we had to set priorities. I explained that the SSC was going to help us learn the laws of nature, and I asked if that didn’t deserve a high priority. I remember every word of his answer. It was “No.”

What does motivate legislators is the immediate economic interests of their constituents. Big laboratories bring jobs and money into their neighborhood, so they attract the active support of legislators from that state, and apathy or hostility from many other members of Congress. Before the Texas site was chosen, a senator told me that at that time there were a hundred senators in favor of the SSC, but that once the site was chosen the number would drop to two. He wasn’t far wrong. We saw several members of Congress change their stand on the SSC after their states were eliminated as possible sites.

I was hoping he would also delve into some alternative models for funding big science. He proposes raising taxes, but that itself is another political battle. Perhaps the X Foundation and its quest to move space exploration forward with private funds is one model.

• Nathaniel Ayewah and William Pugh. 2010. The Google FindBugs fixit. In Proceedings of the 19th international symposium on Software testing and analysis (ISSTA ‘10).

• Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, Dawson Engler. A few billion lines of code later: using static analysis to find bugs in the real world. Communications of the ACM archive Volume 53 , Issue 2 (February 2010) Pages: 66-75.

What both these papers have in common is that they document the experience of computer science academics applying their tools to real, and large, code bases. The first one talks about Bill Pugh’s sabbatical at Google, which he spent applying his FindBugs tool to the massive mountain of Java code there. The second describes Dawson Engler’s experience commercializing his group’s research in extensible static analysis for bug-finding.

If you are curious about the details, you should just read the whole papers, but these are some of the things that stood out to me:

• FindBugs did find a large number of bugs, but only very few of them were serious or caused production problems. The authors attribute this to the numerous other “filters” that code has to go through at Google: code review, unit tests, integration tasks, canary deployments etc. The major value of regularly using such a tool would be to filter out some of these bugs earlier, as opposed to the usual selling pitch for static analysis tools, which is to avoid catastrophic bugs.
• Both tools are unsound: they make their best effort to find genuine problems, but might also report false positives. This is a major departure from most academic research, which looks down on unsoundness.
• In the case of Coverity, I found it very amusing to read how and why their tool had to be essentially dumbed down, because complex bug reports that were genuine but not understood by developers were simply dismissed as false positives. They also have a number of juicy quotes from clueless programmers.
• There are two broad kinds of properties of code that tools can try to infer. Liveness properties say that the code is good: does what its supposed to, makes progress etc. Safety properties speak about the absence of bad things: no null pointer references, out-of-bound reads etc. Both these tools, as well as every other one I know of, completely give up on liveness and focus on safety. This makes sense, because, in general, liveness properties are much harder to prove, and also require programmers to provide a rigorous description of what the code is supposed to do.
• These tools are context-insensitive: they examine a piece of code locally, without taking into consideration all the paths that could lead there. Context-sensitive analysis is too hard and explodes combinatorially for large codebases.

The cost of a bug

In this context, it is useful to ask: what is the cost of a bug? Sure, we would like to eliminate every bug if possible, but what are the tradeoffs?

Let’s take a serious class of bugs: those that crash your application. Not quite as serious as something that destroys data, but something in the middle, and that happens frequently.

The question then becomes: what is the cost of a crash? This is where the economics become interesting.

For client applications, the cost of a crash is pretty high. A crash makes your users unhappy and complain and leave, and justifiably so, because the application they were using just crashed on them! On top of that, you do not control the environment in which your application runs, and have to bend over backwards to get crash information to help you debug.

Things are very different for server-side applications served over the web. They are typically request-oriented, served by lots of instances. If one of them crashes, and the application stack is smart enough with retries, there is no user impact at all. What’s more, you control the environment the crash ocurred in, and have all the debugging and trace information you need. And you can push a fixed binary at your pleasure without anybody even noticing.

This means that the cost of a crash is much, much higher for client software than for server software.

The future of bug-finding

The major roadblock for no-holds-barred context-sensitive analysis of code is the exponential blowup in the space being analyzed. But this is only a hurdle if you are holding on to the single-machine model of building and analyzing code, i.e., that you have one machine (albeit a monster) that builds your code and tests it and analyzes it.

But what if you could throw a cluster of a few hundred machines at the problem, and have them running continuously? How much deeper and fine-grained could your analysis be?

I love that lambda support in Python is great enough to be able to do this.

Church numerals are an encoding of the natural numbers in pure lambda calculus. It’s fascinating to see how you can build up everything from this bare simplicity.

The code is here, and reproduced below.

#! /usr/bin/python
#
# Church numerals in Python.
# See http://en.wikipedia.org/wiki/Church_encoding
#
# Vivek Haldar <vh@vivekhaldar.com>
#
# https://gist.github.com/2438498

zero = lambda f: lambda x: x

# Compute the successor of a Church numeral, n.
# Apply function one more time.
succ = (lambda n: lambda f: lambda x:
        f(n(f)(x)))

one = succ(zero)

# Add two Church numerals, n, m.
add = (lambda m: lambda n: lambda f: lambda x:
       n(f)(m(f)(x)))

# Multiply two Church numerals, n, m.
mult = (lambda m: lambda n:
        lambda f: lambda x: n(m(f))(x))

# Exponentiation: n^m
exp = lambda m: lambda n: n(m)

# Convert a Church numeral into a concrete integer.
# n = 0, f = (add 1 to a number)
plus1 = lambda x: x + 1
church2int = lambda n: n(plus1)(0)

# Convert a concrete integer into a church numeral.
def int2church(i):
    if i == 0:
        return zero
    else:
        return succ(int2church(i - 1))

# Eval and print a string
def peval(s):
    print s, ' = ', eval(s)

peval('church2int(zero)')
peval('church2int(succ(zero))')
peval('church2int(one)')
peval('church2int(succ(one))')
peval('church2int(succ(succ(one)))')
peval('church2int(succ(succ(succ(one))))')
peval('church2int(add(one)(succ(one)))')
peval('church2int(add(succ(one))(succ(one)))')
peval('church2int(mult(succ(one))(succ(one)))')
peval('church2int(exp(succ(one))(succ(one)))')
peval('church2int(int2church(0))')
peval('church2int(int2church(1))')
peval('church2int(int2church(111))')

c232 = int2church(232)
c421 = int2church(421)
peval('church2int(mult(c232)(c421))')
print '232 * 421 = ', 232 * 421


c2 = int2church(2)
c10 = int2church(10)
peval('church2int(exp(c2)(c10))')
print '2 ** 10 = ', 2 ** 10

Heavy-duty messengers, such as railways, devised their own intricate codes with short forms for messages they commonly needed to communicate. These codes were then spread throughout the organization, to be used by both telegraph senders and receivers. And telegraph operators tried their best to outlaw the use of such codes.

That was an early form of the same carrier-messenger war we are currently seeing with issues such as net-neutrality. Carriers want to shape the content on their pipes to suit their business needs. And messengers want the carriers to not care about their messages and never look at what is running through the pipes.

For journalism the package was the newspaper, a collection of articles. For colleges, the package is the degree, a collection of courses.

Of course, with Khan Academy and Udacity and OCW and many others, the courses are free. This, among others, is a major part of the argument of growing anti-college brigade who say that you’re better off not going to college, saving your money and not getting saddled with debt, and instead getting work experience for those four years, perhaps moonlighting with online courses.

Degrees

Colleges retort with: “we certify you! A degree is a stamp of some sort of minimum competence.”

But on the demand side, employers (at least in software) have almost completely soured on degrees. Their hiring decisions are based completely on how well you do on the interview, not on what degree you got from which college. And the interview focuses completely on the thing you’re being hired to do (e.g. programming). You would be amazed how many people with CS degrees simply cannot write programs.

Cost structure

The thing is: colleges are stuck in a death spiral of costs increasing, and increasing costs turning off applicants.

Why are costs high? Because it takes a lot of people, and they have salaries and health benefits, and retirement plans. But also because of the resortification of colleges. Larger fractions of funds are going towards building lavish facilities and buildings and recreation centers and dorms that are not dorms but fully furnished apartments. Those are now the baseline for attracting applicants.

Return to basics

So what will become of colleges?

I like to expand the question to encompass the thing that contains the college: the university. University and college have been synonymous for a long time, but a college is only one component of a university, which is a much broader institution.

It not only disseminates knowledge, but also creates and curates it. It’s the dissemination part that is being unbundled and dis-intermediated. That is the part that can be put online, for free.

The fact still remains though, that colleges survive on the money from undergraduate tuitions. And that is exactly the part that is beginning to go away.

Just like newspapers, the money from new sources of revenue will never come up to the level from the sources that are going away.

So what gives?

Remember that universities started out as essentially learning monasteries. You went there if you had a thirst for knowledge, but there wasn’t a pre-packaged “degree” to be had.

Likely, universities will go back to their smaller, simpler roots, concentrating more on unstructured knowledge creation.

Contrast this snippet…

Those same iPhones illuminate the Garden like the cigarette lighters of yesteryear. There is the real Kanye West on the stage, with a giant close-up of his sweaty face projected above, and there are thousands of tiny Kanyes dancing and rapping in the palms of the audience: it’s the Watch the Phone tour, I realize, a weird, technologically mediated experience that makes the concert at once more intimate and more distant.

… with this one, taken from Stephenson’s “In the beginning was the command line”, written in 1999:

I was in Disney World recently, specifically the part of it called the Magic Kingdom, walking up Main Street USA. This is a perfect gingerbready Victorian small town that culminates in a Disney castle. It was very crowded; we shuffled rather than walked. Directly in front of me was a man with a camcorder. It was one of the new breed of camcorders where instead of peering through a viewfinder you gaze at a flat-panel color screen about the size of a playing card, which televises live coverage of whatever the camcorder is seeing. He was holding the appliance close to his face, so that it obstructed his view. Rather than go see a real small town for free, he had paid money to see a pretend one, and rather than see it with the naked eye he was watching it on television.

This comes as a huge surprise to everyone except the scientists, grad students and post-docs who actually performed these experiments and wrote them up.

This is not failure of ethics. Scientists are not trying to publish fabricated results. It is a failure of process and documentation. If you teleported into the lab while the experiment was being done, you would actually see the same results that were written up.

But then going to another lab and trying to set it up just so, and get the exact same result? Good luck with that.

Ask any grad student and they’ll tell you just how delicate these setups are. They work for months just to get a stable setup, and then when it works, they try not to breathe too hard on it and pump it for as much data as possible and publish a paper. They aren’t going for reproducability, just producability.

Why?

Because the paper is the ultimate goal of all this. Grad students need them to graduate with a PhD. Professors need them for tenure and funding. The department needs them for prestige.

Like I’ve argued before, the paper as a unit of dessimination of modern scientific results is outmoded.

A paper doesn’t even contain all the information and data required to reproduce a result. Because if it did, it would be the size of a book. It usually has just the high-level points and a summary of the data.

Instead, what if the paper was replaced by a publicly-visible activity log? A long series of small steps, each one small enough to understand and debug and… reproduce.

Every time I sit down to think something through with pen and paper, I make so much more progress and have so much more clarity of thought, in a much shorter amount of time, compared to using any electronic device.

The downside, of course, is that paper is analog. It has life separate from your digital universe, which by now is most of your universe.

The solution isn’t to make screens and keyboards behave more like paper. The better way to go is to think of ways to make the act of moving a pen–like instrument across a flat surface something that can be digitized.

One way to do that would be to make the surface, the “paper”, digital. This is the approach that Windows tablet PCs took in the last decade. You could use a stylus to write on the screen itself. The handwriting recognition was smart enough that your notes could be searched just like text, perfectly integrating them into your digital world.

Unfortunately, Tablet PCs faded, largely because they treated the stylus as a poor 2nd class citizen, shoehorning it into the regular mouse and pointer and clicking based Windows UI, without trying to make an interface that felt more natural with a pen. Also, tablet PCs were basically just Windows laptops with capacitive screens–not the lightest and easiest things to carry around.

Another approach is to make the instrument, the pen, digital. This is what Livescribe has done. You use one of their pens to write on regular paper¹, and it captures what you wrote. Later, you can connect the pen to your computer, and get an electronic copy of everything you wrote or doodled.

Having an electronic pen and using it with analog paper has a bunch of advantages over having an electronic screen. When you’re actually writing, paper is a lot more convenient than the screen. Also, the pen is much smaller and easier to carry around. I’ve been using this for about a year, and I love it.

The role of code review at Google has been pretty well-documented by now. It is absolutely central to the way it builds and operates engineering systems. There is a small river of posts by programmers about how great code review is. And I’m going to add to that, because I don’t think is is something that can be stressed enough.

No matter how great or experienced a programmer you are, code review will always improve your code.

The sad truth is that you are blind to your own code. Once it compiles and runs and does what you want it to do, you don’t want to go over it agian. “It runs! Ship it!” No. A fresh pair of eyes will tell you exactly what you missed. And you will go, “Doh! Shoulda fixed that.”

If dubious interpretations of scientific data appeared only once in Imagine, it might be a worrisome fluke; but they appear multiple times, which is cause for real concern. Lehrer steps over the line again when connecting amphetamine use to creativity. He states that “Because the dopamine neurons in the midbrain are excited…the world is suddenly saturated with intensely interesting ideas.” Such definitive statements imply that neuroscience has already charted a causal course from neurotransmitter chemistry to a complex cognitive process — which simply isn’t true. That it should have come from a writer who so clearly has the ability to write about science critically and intelligently still comes as a bit of a surprise.

We need good translators of science to the general public, and Lehrer has the public’s ear and the public’s trust. He is at his best when putting his considerable talents to the task of telling a story that is true according to the facts as we know them, rather than telling a story people want to hear.

Lehrer actually studied neuroscience in grad school and is fully capable of grasping the technical nuances of the research in the field. This is why practicing scientists shy away from writing “popular science”. They are scared of giving up the fine chisel of the lab for the broad strokes of prose.

Coding is not the new literacy because it will never be a requirement that every man, woman and child must know how to code in order to communicate fully. It is a function of the development of communication, not its application. As such, it is more true to compare coding to skills such as book-binding, or newspaper printing. It is sufficient for a relatively small proportion of the population to understand it in order for its benefits to accrue.

I think this is a shortsighted view to take. What qualifies as a “literacy” has drastically changed throughout history. The ancient Greeks considered oratory skills such as rhetoric and debating to be the foundation of their education. When writing came along, they resisted, thinking that committing thoughts to tablets would weaken their minds.

At the same time, I do realize that “coding is the new literacy” has become a cliché. But if you take a broader view, and think about not coding per se, but the thought processes around it, the concept becomes much more powerful. Thinking computationally, even about problems and domains that don’t involve coding, is rapidly becoming an essential skill, and a mode of thinking that one needs to be familiar with to understand the modern world around us.

Jeanette Wing captures this reasoning perfectly in her description of computational thinking:

Computational thinking is a fundamental skill for everyone, not just for computer scientists. To reading, writing, and arithmetic, we should add computational thinking to every child’s analytical ability. We have witnessed the influence of computational thinking on other disciplines. For example, machine learning has transformed statistics. Statistical learning is being used for problems on a scale, in terms of both data size and dimension, unimaginable only a few years ago… Computational biology is changing the way biologists think. Similarly, computational game theory is changing the way economists think; nanocomputing, the way chemists think; and quantum computing, the way physicists think. This kind of thinking will be part of the skill set of not only other scientists but of everyone else.

And another thing I’m fascinated by is language and etymology.

• English words lifted from Hindi
• Nouns as verbs, verbs as nouns
• The 18th Century Origins of Lolcat-speak
• Parsing “and/or”
• Englsh Wtht Vwls
• Upsight

Or is it possible he just loves programming? No doubt the filmmakers considered this option, but you can see their dilemma: how to convey the pleasure of programming—if such a pleasure exists—in a way that is both cinematic and comprehensible? Movies are notoriously bad at showing the pleasures and rigors of art-making, even when the medium is familiar.

(From Zadie Smith’s essay, Generation Why)

This is a problem that programmers often face. How exactly can they convey the sheer awesomeness of the stuff they create? Painters can point to their paintings. Mechanical engineers can point to shiny metallic objects. What can programmers show? Pages of incomprehensible code?

This problem is all the more exacerbated for programmers deep in the backend–those who hack infrastructure. At least programmers closer to the product and the front-end can point to shiny UIs and things happening on the screen and say “I made that!”

Another theme is the economics of programming:

• Why developers should learn the economics of code
• The 0.1x developer
• The CS assignment I wish I had
• Innovator’s dilemma in programming languages

One of the more prominent themes is the nature of modern work, and how it relates to Taylorism:

• Taylorism in the modern tech industry
• Engineering is all about failure
• Minimalism is not a viable intellectual strategy
• The programming assembly line

There’s another interesting reason for using a specific language, but it becomes a factor very rarely – indirect competition. This is when company chooses and promotes a language primarily to beat back a competitor.

This happened in the 90s when IBM decided to throw all its weight behind Java. Microsoft was ascendant, and so was its developer stack. IBM was terrified that it would end up in a position where it would have to use (and license!!) Microsoft’s languages and compilers, and be shackled by the APIs of Redmond.

Along came Java, that too from a company whose CEO at the time was a sworn enemy of Redmond. It was a godsend for IBM. It made a strategic decision to throw all its weight behind Java.

Java’s rise, particularly inside the enterprise, had a lot to do with Big Blue backing it over the next decade.

What is the job that programming languages are hired to do?

Here is a partial list of criteria for picking a programming language:

• Technical: the language is superior along some technical dimension. It might have great support for parallel message passing applications, for example. Unfortunately, most programming language debates stay stuck here.
• Economic: you run large applications on humongous numbers of machines, which cost a gargantuan amount of money. You would very much like to not waste any of that hardware, utilize it as much as possible, and reduce the marginal expenditure when the size of your application, or the number of its users, increases.
• Business: your code must react very quickly to changing markets, and new customer requirements. Maybe the requirements are not clear, and the only way to learn is to release something, see how it does, and then adjust your product. Agility, and reducing the time it takes to implement a change, are of paramount importance, or else you will simply be squeezed out by more nimble players.

A large company with acres of data centers and a small startup renting VMs over the net will “hire” languages to perform very different jobs.

Let me propose a new dividing line for programming languages:

• Sustaining languages: these are the ones commonly thought of as “large-scale industrial languages.” They stress efficient use of hardware resources (CPU, RAM) over personal productivity and succintness. They are typically used for “stable” applications, where the market and requirements are well-known and slow-changing¹. They have been around for a long time. However, a continuous stream of incremental improvements has sustained their position.
• Disruptive languages: the new kids and upstarts. They stress agility of expression, succintness and individual productivity and expression, and are willing to use resources inefficiently to do so. They are usually looked down upon by the establishment (i.e. those who work with sustaining languages). They address a very different market. They are typically used at small scale, and for products whose requirements are not yet known, and will only be known by releasing something, getting user feedback, and iterating.

Currently, C++ and Java are examples of the first, and Python and Ruby are examples of the second.

It is no surprise that C++ and Java came out of large companies, and Python and Ruby came out of individuals scratching an itch.

I call them disruptive languages because like Christensen’s disruptive technologies they address a market not catered by the establishment, and are considered to address the “lower end” of the market.

But disruptive technologies are disruptive because they move “upmarket” and dislodge the incumbents. How might that happen with languages?

What is the tipping point at which a small company should consider re-engineering its software? In other words, when should it stop spending hardware on people, and start spending people on hardware? That starts to happen when hardware costs begin to approach people costs. For a small startup, that might happen when hardware costs cross the annual salary of one developer (or a few). If you look at AWS prices, you’ll see that you can rent a lot of hardware for that much.

But the cost of hardware has a clear trajectory over time – downwards. This means that the afore-mentioned tipping point will get pushed further over time. That in turn means that the scale (size of data, number of users, or whatever metric is relevant) up to which you can economically use disruptive languages is increasing over time.

Me: Hmmmmm… what’s in it for me?

Recruiter: A special t-shirt.

Me: Cool. I’ll do it.

Actually, the “what’s in it for me” is not spoken out loud. It’s an implicit contract.

Time and again, I’ve seen myself and other people go to great lengths to earn a t-shirt. Above and beyond the call of duty. More than they would for cold hard cash¹.

Everyone in tech knows exactly what I’m talking about. T-shirts are currency.

Money? Money is common. A dollar is like every other dollar. It’s not special.

A t-shirt says something. It’s visible. You can show it off. It’s an invisible handshake among those who know what the t-shirt symbolizes. A team. An old, dead, but pioneering company. A small event. A place. You can gloat when someone asks “Cool shirt. Where did you get it?”

That’s why their design is taken seriously. People spend time and effort and breath on it.

Think about that next time you want to ask someone for a favor.

CDs had just arrived on the scene, but were totally unaffordable, costing about an order of magnitude more than cassettes. And they were so new that very few people (definitely none of my high school buddies) had CD players, killing any prospect of music sharing.

And so I gradually built up my music collection on cassettes. The thing about cassettes is, they weren’t like CDs, or now, MP3s, where once you had the thing it was trivial to play it.

You see, not only were they analog, they were mechanical. They spun. Things rubbed against each other. They wore out. They got jammed. They got dirt in them. Sometimes, they just plain broke. And God forbid, if you forgot one of them in the car in the summer. They would melt. Sometimes, the trauma was recoverable. Often, it wasn’t. Sometimes I had to transplant the tape from one cassette container into another one, or switch parts among them. And we haven’t even got to the player, with it’s motor, and the tape-reading head that got dirty and had to be cleaned.

With cassettes, you had to be involved. At scale, with enough cassettes and wanting to listen to them for enough hours, getting your precious music to play off cassettes required active coaxing and an ongoing commitment of time and effort. Not unlike tending a garden.

You had to touch them. Tinker with them. Tear them apart. Put them back together. Appease them. But you knew them. Which ones were good. Which one would need some fixing before being played today. Which ones stuck around which songs. Some of them were unlendable because only you knew how to get the music out of them.

They were yours. You were intimate.

I largely agree with her, but have a few observations to add:

• progress seems slow because being incremental is necessary for any hope of wide adoption. The population of practicing programmers, as well as the ocean of existing code, can only absorb so much change at a time. Languages that are too radical don’t make it beyond the paper they’re described in. For example, lambda expressions and closures have only recently made (or are making) the jump from largely academic functional languages to mainstream ones like Python, C++ and Java.
• this is further exacerbated because the window of opportunity for new languages to take off is extremely slim. Languages don’t take off by themselves, they ride on the backs of platforms, and those only come along once or twice a decade.
• the mantle of programming language innovation firmly passed to industry sometime in the 90s. Why? Because that’s where all the interesting problems were starkly revealed. Matt Welsh went from Harvard to Google because he found “problems that are orders of magnitude larger and more interesting than I can work on at any university.” This is not to disparage academic PL research, because a big part of new PL design is cherry-picking the best constructs from past academic research.
• that said, I think currently we’re in the valley of “good enough” languages. There are certainly tons of annoyances and there will always be room for improvement in the mainstays (e.g. Java, C++ and Python, to count category representatives), but by and large they get the job done. I have yet to run into a project where the language was the bottleneck for success. That doesn’t mean that there are not problems on the horizon (concurrency, distributed computing…) that will eventually become painful enough to force a move to new languages (or old languages with new constructs).
• because of the above two points, language engineering resources have mostly been devoted to virtual machine and runtime engineering, because they provide an immediate payback in terms of hardware dollars utilized and saved.

I think there’s a good argument that a large portion of modern science is Taylorist. The myth of the lone genius with a “eureka” moment doesn’t describe the vast majority of science that actually takes place, especially in the more costly areas like physics and biology (mathematics might be different). Labs and universities are run on a quite industrialized basis, with professors typically supervising a whole stable of researchers, agendas driven by grants and metrics such as publication impact factors, and so on. It’s not 100% a factory, but I would say that 90% of it is at least 80% factory-like. Grad students are certainly cogs in a large number of cases.

This is happening without any loss of impact, i.e. team-authored papers have been getting more citations than their solo-authored counterparts, and the gap is widening.

Extrapolating current trends, there will be no new single-author papers in the sciences by approximately 2020.

The data only confirms what most people in the sciences already knew for a long time. It is now completely a team activity. Each player is incredibly specialized. So much so that to write a single paper, which itself is about something highly specialized, requires a team of authors, each of whom has specialized in one little part of that. You could be grinding away in a lab for months, doing experiments, generating data and analyzing it, and you could end up contributing only a few paragraphs.

Not even half a lifetime ago, the giants of science were comparable in stature to the size of their fields. But now, they are more like generals commanding vast armies to climb uphill. They might have the authority and the vision, but nobody, including themselves, believes even for a moment that they could have made their conquests without their armies.

Special/General

So should one specialize or generalize? It’s a question of degree, but one can still pick a direction: to know more and more about less and less, or to sacrifice some depth to gain breadth.

In at least one study², marked gains have been found for specialization. But the study was done done in the context of pro football, and I’m not sure about its applicability to knowledge work.

However, all the macro-level forces are pushing researchers towards specialization. The complexity simply demands it.

But then who has the bigger picture? Whose hands are on the rudder of this giant ship of worker bees? More often than not, that person is a technocrat–a cross between a scientist and a bureaucrat. They are smart and technical enough to understand the work. They are good managers. They are great at selling the work to funding agencies. They are the ones keeping the lights on.

Just like in almost every other field of human endeavor, the middle is gradually being hollowed out. You start out as a lab rat. If you are really good, you will eventually become a technocrat. But if you are only mediocre, you will never be able to jump across the vast chasm between the two.

This is where deep specialization might actually end up hurting you. Because in order to be a technocrat you need to generalize. You need to know how your tiny little field fits into the bigger jigsaw puzzle. You need to convey this to the people with their hands on the purse strings.

HAL, please list the experiments to be carried out

At the end of Kevin Kelly’s talk³ about how science is carried out, someone asked an interesting question: “so we’ve already seen the first thousand-author paper. When do you think we’ll see the first zero-author paper?”

The answer to that question will probably define the nuts and bolts of how science is actually carried out in the future. If the totality of the knowledge in even a highly specialized field is beyond the comprehension of not just one but even a small team of humans, and if all they can hope to do is make a long series of small edits to the giant wiki of science⁴, will we eventually turn to machines to summarize that knowledge into terms that can be readily understood, or going even further, suggest fruitful opportunities for investigation?

The political method, on the other hand, targets believers rather than skeptics. Convincing people to switch is hard, but it is easier to give believers even more reasons to strengthen their belief. Over time, the believers, who are immune to changing their minds, will convert the undecideds.

A recent paper¹ modeled precisely this effect, and backs up this tactic. Even a small fraction (as little as 10%) of committed agents immmune to change can rapidly convert the entire population to their position.

Note that the technical and political method differ not in their use of logic – they both present facts and logic (or at least, what they perceive to be facts and logic) – but in whom they target and how they are targeted.

The students at the Institute believed in the religion of science and reason. The entire modern world around them was built on it. And they were learning the incantations that would one day allow them to be part of the priesthood that creates it and cares for it and carries it forward. The baton of progress and civilization would be passed on to them, and they would run with it. Every (technical) class and textbook that they went through reinforced this belief. Once they learned about the Fourier transform in their signals and systems class, they looked around and were astounded by how much of the modern world rested on that concept. The feeling was repeated with nearly every class they took. They could’t help but feel awe, quickly followed by a tingly sense of power.

And then they had to go to some hookah, where a professor talked about art, or sociology. There were no hard definitions. In fact, the biggest debates were about definitions. E.g. “What is art?” The complexity was like quicksand, all-envoloping, yet not something one could solidly stand on. In fact, the goal was not to arrive at an “answer” but to find out how complex an issue could be.

It was only years (decades?) later that they realized that reason is not everything, that people, not technology, are usually the bottleneck, and that only by knowing the full range of the complexity of an issue can one hope to understand what they’re giving up when the time comes to collapse that complexity down to a single choice.

What is this rule? I’m going to call it the principle of rational equivalence: two rational people, given the same set of facts, must arrive at identical conclusions.

Of course, this is nothing but a fuzzy extension of modus ponens, the very bedrock of logic.

And of course, this is violated all the time in the real world. People see a situation, which presumably is objectively invariant, and yet arrive at completely opposing conclusions.

They are many explanations for the violation of rational equivalence (people are not rational, the facts are incorrectly perceived, or even if people are rational and the facts are correctly percieved, they make mistakes of logic etc etc), but every time it happens it still confounds most rational technical people. It’s a cognitive blind spot. It’s like one of those false proofs of something absurd where you know the result can’t be true, yet every step seems legit.

• I love to write.
• Writing about something forces me to clarify my own understanding of it. Oftena thought rattles around in my head until I make an effort to write it out.
• I hope someone will correct me if I’m wrong, and I’ll learn.
• My blog collects my thoughts about stuff I read.
• I love the pageviews, retweets, likes, +1s. No, I’m not above that, as much as I’d like to be. Ironically, it is the analytics that expose the futility of chasing poopularity. Posts that I shot off often become popular, and posts that tortured me for months often sink without a trace.

The perfect antidote to Carr is Kevin Kelly. His “What Technology Wants” is a thorough defense of the technological enterprise. What makes it special is that it is written largely from the point of view of a skeptic – i.e. he is wary and distrustful of technology, yet ends up favoring it in the final analysis.

A couple of previous posts instigated by Carr:

Taylorism in the modern tech industry

The Internet is making us smarter, but is it making me stupid?

We played a few turns. I was a little down, as expected. Then a funny thing happened. It was my turn, and the board was particularly complicated. It would take something fancy to drop a coin.

My opponent came over to my side of the board, analyzed it, and started suggesting various shots.

“What the hell are you doing? Why are you helping me? I’m playing against you, remember?”

“Who cares who’s playing against whom? I want to make the game more interesting.”

I think a paper in a journal is an antiquated way to desseminate results this complex, created by teams this big.

There is a precedent for a system that can track, manage and publish the work of large teams on a single, though complex, artifact – source control. Particularly, distributed source control systems such as git.

Consider an alternative publishing model based on source control that works as follows:

• all artifacts for the project (or at least, those that can be digitized) must be checked in. Documents. Scripts. Data analysis. Raw data from experiments. Lab notes. Diagrams. Everything.
• every “commit” is subject to rigorous review by your peers. For day-to-day commits this will usually be your immediate team members. For more substantial “commits” or results, external reviewers may be called.

Compared to the existing model of writing papers and submitting them to journals, this model has a number of advantages:

• the progress of the project can be accurately traced over time, by looking at the history of commits.
• questions about results are easily answered. “How did you arrive at this table?” “Let me go look at the commit.” There are no throwaway scripts lying around in people’s home directories that are lost. Every analysis is reproducible.
• individual contributions are clear. Just look at a person’s commit history.
• the process of producing scientific results can be clearly seen. With journal papers, one could only see the end result of a long series of experiments and analysis, rarely getting a glimpse into the meandering path which brought the investigators there. By looking at not just the commits, but the comments made during their review, one can get a deeper insight into that path.
• collaboration across sites and universities is simplified. Ideally you wouldn’t even need permission. Repositories would be world-readable and you could simply “pull” from them. If you spot a problem or have a contribution to make, you don’t even need to be part of that group. Pull, fix, then request a review for your “patch”.

What would a “paper” look like in this world? It would be a high-level journey explaining the big picture of the repository in question, linking copiously to artifacts in it. Almost like a top-level README. The curious would be free to dig into the commits or series of commits they’re more interested in.

What would a “journal” look like in this world? It would be nothing but a link aggregator, pointing to repositories of projects and experiments the editors deem worthy. You could cut editors out altogether, and move to a voting model like Hacker News or Reddit.

The mystery deepens almost by the day, because we’ve now identified 700 extrasolar planets and the count is increasing rapidly. We should shortly be racking up lists of Earthlike worlds, and we’re closing in on good estimates of how many there must be in our galaxy. And the number is in the billions. So one central argument against the existence of alien life–the ‘rare Earth’ argument that environments to host it must be rare–has been more or less disproven. And that, just this year.

He summarises with a quote that is sure to become as iconic as Clarke’s original, on which it is a riff:

…any sufficiently advanced technology is indistinguishable from Nature. Basically, either advanced alien civilizations don’t exist, or we can’t see them because they are indistinguishable from natural systems. … It amounts to saying that the universe provides us with a picture of the ultimate end-point of technological development. In the Great Silence, we see the future of technology, and it lies in achieving greater and greater efficiencies, until our machines approach the thermodynamic equilibria of their environment, and our economics is replaced by an ecology where nothing is wasted. After all, SETI is essentially a search for technological waste products: waste heat, waste light, waste electromagnetic signals. We merely have to posit that successful civilizations don’t produce such waste, and the failure of SETI is explained.

When I first read that the explanation didn’t sit right with me. It took me a while to express why.

There is a school of thought that views bits, not atoms, as the fundamental building block of the universe. The march of civilization is nothing but an evolution towards more and more sophisticated information structures, and deeper and deeper stacks of abstraction. Living organisms are ephemeral, their true purpose being to transport, mutate and advance the information substrate from whence they came.

Whether you agree with this school of thought or not, the trend is indisputable. Civilisation is nothing but a giant, distributed, continously evolving information structure. To lose knowledge is equivalent to going back in time. The fastest growing thing in the world is information.

And central to the information is acting on it, and changing it, and refining it. In other words, computing. Civilization would be unthinkable without intricate and advanced information processing and computation.

So how does that square with Schroeder’s deduction? If civilization ultimately “re-wilds”, where is all the information and computation? Do we give all that up?

That seems like a pretty anti-climactic end to all this.

Then I ran into this paper² that surveys how computation and biology can take inspiration from each other: new algorithms that mimic biological systems, and computational models that help understand biological systems.

I’ve also noticed that large computer systems display organic behavior³.

That in turn led me this paper⁴ which was one of the first in the field of molecular computation. The abstract:

The tools of molecular biology are used to solve an instance of the directed Hamiltonian path problem. A small graph is encoded in molecules of DNA and the “operations” of the computation are performed with standard protocols and enzymes. This experiment demonstrates the feasibility of carrying out computations at the molecular level.

Aha! That opens up a line of progress that is compatible with preserving computation and moving technology closer to nature. We can embed computation in nature, at microscopic levels.

Even better, computation at the molecular level turns out to be potentially extremely enery efficient:

…in principle 1 joule is suffcient for approximately 2 * 10^19 such operations. This is remarkable energy effciency considering that the second law of thermodynamics dictates a theoretical maximum of 34 * 10^19 (irreversible) operations per joule. Existing supercomputers are far less energy effcient; executing at most 10^9 operations per joule.

I feel better now.

With the goal of rectifying that, I wish I had been given the following assignments in some sort of advanced systems course, where people work in teams and the team with the maximum metric for the assignment “wins”:

Offline or batch: sort as many integers as you can in 1 hour on a computer that costs $1000.

Online queries: Build a plain text search and indexing system, and reply to as many queries per second as you can on a computer that costs $1000.

To keep the playing field level, the instructor should set the cost of various components (CPU, disk, RAM, flash, network) to be the same for the entire class. You are free to build your computer as you wish, as long as you stay within budget.

Perhaps you can do better with two computers, each of which costs $500. Go wild.

Every year, the instructor should index the costs against the market, and either make components cheaper, or decrease the overall allowed budget. That way students can compete not just with their own class, but classes of previous years as well. Just for fun, the instructor should maintain an all-time leaderboard.

It is a landmark piece of abstract art. But what made it special was that Matisse documented the evolution of the work from beginning to end, taking photographs of the intermediate phases over the five months he worked on it¹. It started with some realistic depth and detail, and ended up with an abstract representation. And the journey was visible.

And the culmination:

Matisse said²: “Supposing I want to paint the body of a woman: first of all I endow it with grace and charm, but I know that something more than that is necessary. I try to condense the meaning of this body by drawing its essential lines. The charm will then become less apparent at first glance, but in the long run it will begin to emanate from the new image. The image at the same time will be enriched by a wider meaning, a more comprehensively human one …”

Then there’s The Bull, by Picasso ³.

It starts like this…

… evolves to this…

… and this …

and ends up at:

In both these cases, you can see the exploration, but with a clear direction. You can behold not just the work of art, but how it got to be what it is.

Thanks for reading!

• Size is the best predictor of code quality
• The Cognitive Style of Unix
• GUI vs. CLI: Operation vs. Expression
• Smeed’s Law for Programming
• Stallman’s dystopia
• Minimalism is not a viable intellectual strategy
• Editing Google Docs in Emacs
• OOP = FP?

Also: Best of 2010

1. Be incremental. Your new programming language must be understandable in terms of concepts that most programmers already understand. If you are going into new territory, you must justify it with shortcomings in the incumbents.

   C, which was probably the biggest jump we have ever seen in terms of how different it was from other languages of the time, i.e. assembly specific to your hardware architecture, was all the same close enough to the metal that one could conceptualize the assembly code corresponding to a snippet of C code.

   C++ grafted the concept of classes and objects on to C in a manner that would provide the gentlest on-ramp for C programmers. In fact, the earliest C++ compilers did not emit machine code, but simply compiled C++ down to C, and then used the C compiler.

   Repeating the same script, Java adopted a syntax that would make C and C++ programmers feel right at home. By the time it was introduced, most programmers were comfortable with object-oriented concepts.

2. Ride a platform. Languages are incidental. What people really adopt are platforms. So if you want a language to be successful, you should tie it to a rising platform early in both their lives.

   C got popular on the back of UNIX.

   JavaScript rode the browser and the web.

   Ruby got popular because people wanted Rails.

3. Get taught in computer science courses. It really helps when the next generation of programmers are already familiar with your language when they graduate.

These are all necessary, but not sufficient conditions. If you look at the long list of programming languages which have failed to achieve mass usage, you will see that they did not satisfy one or more of the above.

PS: what counts as success for a programming language is itself complex and open to much debate. For the purposes of this article, I equate success with mass usage and popularity.

It is a fallacy. Not in the sense that certain developers aren’t 10 times as productive as the average, but in the way they achieve that productivity.

People often think that the 10x developer can churn out tons of code in short periods of time, or build massive systems from scratch. That might be true, but it is a side effect, not the root cause of their productivity. These days, if you auto complete 10 times in a decent Java IDE, you can end up with a few hundred lines of code. In fact, seasoned developers often look at a high rate of writing code with suspicion, because the number of bugs you write goes up with the number of lines of code you write²³.

10x developers achieve their productivity by focusing on the 0.1x of the solution that will have the greatest impact. What makes them special is their ability to figure out what that one tenth is. And then blind out everything else. This usually requires a deep understanding not just of the technical aspects of the solution, but also the underlying business problems⁴.

So, recruiters should actually be looking for 0.1x developers, not 10x developers.

66 institutions have had an unbroken existence from the 16th century to the present day, and of those, 62 are universities². This is an astonishing fact given that universities wield almost no power, and compared to other institutions such as parliaments or corporations, have very little impact in the short term. This longevity can only be explained by the fact that we as a society recognize at some deep, unstated level that universities are crucial and worth preserving.

But crucial to what, and how?

In the ancient world, Buddhist monks would go from door to door asking for alms. They were called bhikku, or bhikshu ³, which stems from the Sanskrit root for “beg” (bhik). This pattern is repeated in all major religions. All have a monastic tradition, and one that usually survives on charity.

The earliest universities were nothing but learning monasteries⁴. From the dawn of civilization society has been paying a “monk tax”, supporting those who have to some extent checked out from the turbulence of day-to-day life to focus deeply on something for the longer-term.

In this sense, universities are like the seed corn of civilization. Eating your seed corn might fill you now, but will surely lead to famine in the future.

And now we have come full circle. In the modern world, if you want to check out from the turbulence of day-to-day life to focus deeply on something, but not in a religious way, and do so surrounded by a few kindred souls, the place to do that is a university.

At their core, universities are modern, secular monasteries.

The book is a philosophical meditation on the concept of work. What makes work meaningful, satisfying and enjoyable? What makes it dull, boring and soul crushing? He talks about how blue-collar work can be intellectually challenging and satisfying, and how white-collar “knowledge work” can be repetitive and stultifying. Whether or not you agree with his conclusions, his examination of the underlying issues is stimulating, and is peppered with colorful and entertaining episodes from his own life. He was the director of a Washington think tank, but ended up as the proprietor of a bike repair shop specializing in Japanese motor bikes. So he speaks from experience on both sides of the fence, which already gives him more credibility than most authors on the topic. If you want a shorter introduction to some of the core issues discussed in the book, read these two² essays³ by him.

The primary culprit for the dehumanization of work, according to Crawford, is Taylorism. So we must look at that first, and then we will loop back to what all this has to do with programmers and programming.

Taylorism

Taylorism is so deeply ingrained in every aspect of not just modern commerce but life in general that it is almost impossible to identify it. Just like a fish doesn’t even know that it is in water. If you haven’t read his “Principles of Scientific Management”⁴, it is a short and easy read, and now available free on Project Gutenberg. After reading it, you will realize that it is one of the founding documents of modern life, as well as the progenitor of the “business book” category.

Briefly, Taylorism has two central tenets:

1. Measurement: associate metrics with all aspects of work.
2. The separation of thinking and doing: An educated class of managers measures and plans all the work, and is responsible for the overall process, while a class of laborers carries out the implementation of those plans.

Taylor was convinced that the laborers he was studying were “soldiering”–knowingly working slowly to maintain work and wages for all their brethren.

Consider this dialogue that Taylor had with a worker in order to convince him to move more pig iron:

“Schmidt, are you a high-priced man?”

“Vell, I don’t know vat you mean.”

“Oh, come now, you answer my questions. what I want to find out is whether you are a high-priced man or one of these cheap fellows here. What I want to find out is whether you want to earn $1.85 a day or whether you are satisfied with $1.15, just the same as all those cheap fellows are getting.”

“Did I vant $1.85 a day? Vas dot a high-priced man? Vell, yes, I vas a high-priced man.”

“Well, if you are a high-priced man, you will load that pig iron on that car to-morrow for $1.85. Now do wake up and answer my question. Tell me whether you are a high-priced man or not.”

“Vell–did I got $1.85 for loading dot pig iron on dot car to-morrow?”

“Yes, of course you do, and you get $1.85 for loading a pile like that every day right through the year. That is what a high-priced man does, and you know it just as well as I do.”

“Vell, den, I vas a high-priced man.”

“Now, hold on, hold on. You know just as well as I do that a high-priced man has to do exactly as he’s told from morning till night. Well, if you are a high-priced man, you will do exactly as this man tells you to-morrow, from morning till night. When he tells you to pick up a pig and walk, you pick it up and you walk, and when he tells you to sit down and rest, you sit down. You do that right straight through the day. And what’s more, no back talk. Now a high-priced man does just what he’s told to do, and no back talk. Do you understand that? When this man tells you to walk, you walk; when he tells you to sit down, you sit down, and you don’t talk back at him. Now you come on to work here to-morrow morning and I’ll know before night whether you are really a high-priced man or not.”

(emphasis mine)

The net effect of all this was that the worker could, under the appropriate supervision, now load 47 tons of iron a day, up from his previous 12 ½ tons a day. He got a 60% pay raise, while the company moving the iron decreased its labor per ton by about the same percentage (from 9.2 c/ton to 3.9 c/ton). But another windfall for the company was the saving in time. Previously, moving 100 tons using 2 workers would have taken 4 days, whereas now it would take just a little over 1 day.

The assembly line

The earliest well-known enterprise to go whole-hog Taylorist was Ford, with its assembly line for manufacturing cars. The world never looked back.

The assembly line with narrowly specialized roles for each worker is a consequence of the thinking/doing separation. The smarts are in the process, the assembly line. The workers operating it are fungible. Of course, the workers did not like this at all. Skilled craftsmen that built entire artifacts were replaced with machines that required only semi-skilled operators churning out one tiny part of the whole. The pace of their work was determined by the speed of the assembly-line, which was set by managers. They had lost all autonomy and control over their work. Their work had been de-skilled. The transfer of power from skilled, knowledgeable craft workers to “the management” was sealed. Ford made this palatable by paying his workers high wages.⁵

In his book, Dan Pink⁶ talks about the three factors that make work meaningful: autonomy, mastery and purpose. The fungible non-distinguishable worker manning the assembly line was stripped of all three.

The drive to specialize

The seeds of Taylorism go back to 1817, when David Ricardo outlined the principle of comparitive advantage. He explained how two parties could gain by making goods they were efficient at and then trading with each other, even if each of them could make the traded good on its own. So even if the other party is worse than you at making something, you are better off trading and focusing on your specialty, because of the saved opportunity costs. (See the Wikipedia article for a number of concrete examples⁷).

The undeniable math of comparitive advantage is the natural driver for specialization. Taylor brought the concept down from the level of countries to individual workers.

What does this have to do with writing software?

The field of software engineering was built on firmly Taylorist principles. It had no other choice, because that was the default school of thought.

In 1968, NATO called a conference on the subject of “software engineering” (the field did not have a name back then, and NATO’s pick of terminology stuck)⁸. The industry was having a lot of trouble creating large and complex software, and it was about time something was done about it.

Doug McIlroy expressed what became the rallying cry for the field: “The software industry is not industrialized”!⁹ He said:

Software production today appears in the scale of industrialization somewhere below the more backward construction industries. I think its proper place is considerably higher, and would like to investigate the prospects for mass-production techniques in software. In the phrase “mass production techniques,” my emphasis is on “techniques” and not on mass production plain. Of course mass production, in the sense of limitless replication of a prototype, is trivial for software. But certain ideas from industrial technique I claim are relevant. The idea of subassemblies carries over directly and is well exploited. The idea of interchangeable parts corresponds roughly to our term `modularity,’ and is fitfully respected. The idea of machine tools has an analogue in assembly programs and compilers. Yet this fragile analogy is belied when we seek for analogues of other tangible symbols of mass production. There do not exist manufacturers of standard parts, much less catalogues of standard parts. One may not order parts to individual specifications of size, ruggedness, speed, capacity, precision or character set.

More than 40 years later that still remains the Holy Grail of software engineering.

McIlroy envisioned a world where we would be constructing software by picking components off a shelf, and snapping them together like Legos. The hard work would be building the right blocks, and then it would be easy to snap them together.

Later, Osterweil played a neat recursive trick and argued that the process of making software could be expressed as software too¹⁰. The implication was that it could be executed, just like software.

In some ways this vision has come true, in other ways, it hasn’t.

The biggest example of how the dream of software components has come to pass is web applications. All the components are quite well-understood and, by now, standard. The database. The object-relational mapping. The web server. The Javascript frontend. The webapp developer can pick and choose from a wide variety of choices for each of those. In fact, some grumble that the number of choices adds complexity. All the same, unless you’re building something that must scale to the planet (like Google, Amazon etc), it is ridiculously easy to cobble together a web application that meets 80% of your vision within a day.

Post-modern programming¹¹ starts with Googling, and ends with glue code.

In fact, this componentization has come far enough that many developers complain that the plumbing of pre-existing parts has drained the joy and skill out of their profession¹². Does that sound familiar? The craftsman-turned-assembly-line worker of the 20th century had exactly the same lament!

John D. Cook argues that calling somebody “plumber programmer”¹³ is actually a compliment, because it’s the plumbing and connecting and impedance matching that is hard. But I suspect today that’s true because it’s only the plumbing that is left to do.

How many times can you build a cookie-cutter CRUD app with a web frontend that fetches and displays data from a SQL backend before you feel like all you are doing is cranking a lever? If your Java IDE generates half your code for you, and you auto-complete (Ctrl-Space!) most of the rest, do you feel like you are operating machinery, and not really engaging in expression?¹⁴

The pyramid

In other ways, McIlroy’s vision is still a dream.

Massive development effort still goes into applications that are young and not standardized. There are no components that will let you build your own web-scale search engine quickly, for example.

Programmers are certainly not fungible assembly-line operators¹⁵.

As in all things in life, there is a hierarchy to these things. There is a small number of elite wizard programmers who do write significant things from scratch. Think of those who have implemented language compilers and runtimes, database engines, search engines, operating system kernels and the like. Unfortunately, such projects come along rarely, and the bar for contribution is extremely high.

(By the way, if you are tired of plumbing together databases and PHP and really want to experience the wild West of programming, go contribute patches to the Linux kernel. No fancy languages or libraries that make you soft in userspace. Just plain old C.)

Some problems, such as building webapps backed by databases, are well understood, and their solutions have become templatized. Others, such as indexing a billion documents, or storing a bazillion petabytes across a gajillion machines, are at the edge of our knowledge, and their solutions are still being actively explored.

If you think you are writing cookie-cutter code, it is because you are working on a cookie-cutter problem.

Gratitude

But we must all realize that at the end of the day, we work on something that is in economic demand (in other words, software engineers are well-paid), intellectually challenging (at least, on most days) and in some small way improves the world and helps people. Even all this self-examination and criticism is a luxury that many do not have.

So, be grateful.

But one tradeoff that I think is widely known in folklore but not widely taught is the abstraction-optimization tradeoff.

Before I try to explain what it is, let me give two examples.

1. Compilation to a portable format, and its execution: Let’s take the concrete case of the Java language, and the Java Virtual Machine. The two are distinct. A compiler transforms your pretty high-level Java source code into Java bytecode, which is essentially portable assembly language¹. Now bytecode is still at a higher level of abstraction than the real assembly language of your hardware architecture, which is why you need a Java Virtual Machine to further compile that bytecode into fast, efficient machine code.

Java source has rich structure. It can be represented by an abstract syntax tree, which is the first thing the compiler does with the source code. The process of conversion to bytecode is the process of whittling down that structure to something simple and flat. Bytecode is simple, flat and structureless compared to the source that it came from.

The rules bytecode must adhere to are much more lax compared to those for Java source. In fact, you can write bytecode which has no corresponding legal Java source². This is the reason why the JVM needs to verify bytecode before executing it³.

When it’s time to execute bytecode and further compile it down to the machine code for the real hardware its running on, the very first thing the JVM does is build a control flow graph⁴, which is a feeble attempt at recovering some of that high-level structure of the original source code that was lost in the process of compiling down to bytecode. This is a prerequisite for almost every compiler optimization. Most optimizations depend on teasing out high-level structure from the bytecode.

We started with something with rich structure (source code), compiled it down to something flat and structureless (bytecode), and then had to infer some of that lost structure when it was time to execute and optimize bytecode.

The astute reader might suggest at this point: why not simply make the portable format, the bytecode, something that is higher-level and encodes more structure? I spent a couple of years in grad school exploring that route⁵⁶. It had some compelling advantages over bytecode – it was safe, much more compact, and had the high-level structure necessary for optimization. But it was a non-starter for the “real” world because such a high-level encoding was too coupled to the Java language. If you wanted a portable format for another language, you would have to build a new encoding.

The upside of having a low-level bytecode format is that it decouples the source from the bytecode. This is what allows multiple languages to compile down into bytecode and inter-operate (Jython, JRuby etc).

And that is exactly the tradeoff I’m getting at.

But first, one more example. Because two points are all I need to fit my curve.

2. Effectively Caching IO in an operating system: A similar effect takes place in the caching and IO sub-systems of operating system kernels. Operating systems cache frequently accessed data in memory, to avoid hitting disk as much as possible. This cache is critical for achieving anything like reasonable performance. But the operating system has very little information to go on. It doesn’t know which data is going to get accessed next. So most use a least-recently-used strategy as a next best guess. The longer data in the cache remains un-accessed, the likelier it is to get pushed out. And this usually works pretty well.

So who does know more about data access patterns? The application! And it can tell the OS what kinds of patterns to expect, using “hinting” mechanisms like the POSIX fadvise⁷ call.

Again we see the same pattern. Higher levels in the stack have richer and more structured knowledge, most of which is lost as it travels down to the lower layers of the stack, but that very knowledge turns out to be critical when optimizing for performance. In this case, the application has knowledge of IO access patterns, which is lost in the OS kernel, but is critical for achieving decent IO performance.

Exokernels⁸ was an attempt to solve this problem. As they put it, “An exokernel eliminates the notion that an operating system should provide abstractions on which applications are built. Instead, it concentrates solely on securely multiplexing the raw hardware: from basic hardware primitives, application-level libraries and servers can directly implement traditional operating system abstractions, specialized for appropriateness and speed. ” But with great performance comes a great price, and that price is that they have done away with abstraction. Every time you write an application, you have to pick and choose the OS primitives you want, and think about how you want to wire them together. That’s a big burden.

With the examples behind us, let me make another attempt at phrasing the tradeoff:

The abstraction-optimization tradeoff is between building upon layers of abstraction and achieving optimal performance. Building upon abstraction is often quicker and results in easier to understand solutions, but they might have sub-optimal performance. Sacrificing abstraction leads to near-optimal performance, but at the cost of a more complex solution. The underlying assumption is that richer, more structured information about the thing being optimized (code, time, space) exists at higher layers in the abstraction stack, is lost in the lower layers, yet is critical for achieving optimal performance.

What stood out to me was how the culture of those who work in the Valley is essentially unchanged from what Noyce created half a century ago.

People who run even the newest companies in the Valley repeat Noycisms with conviction and with relish. The young CEOs all say: “Datadyne is not a corporation, it’s a culture, ” or “Cybernetek is not a corporation, it’s a society, ” or “Honey Bear’s assets”? the latest vogue is for down-home nontech names?“Honey Bear’s assets aren’t hardware, they’re the software of the three thousand souls who work here.” They talk about the soul and spiritual vision as if it were the most natural subject in the world for a well-run company to be concerned about.

And at the root of all that was Noyce’s hatred of the class system in corporations from the East Coast, then dominant.

Noyce realized how much he detested the eastern corporate system of class and status with its endless gradations, topped off by the CEOs and vice-presidents who conducted their daily lives as if they were a corporate court and aristocracy. He rejected the idea of a social hierarchy at Fairchild. Not only would there be no limousines and chauffeurs, there would not even be any reserved parking places. Work began at eight A.M. for one and all, and it would be first come, first served, in the parking lot, for Noyce, Gordon Moore, Jean Hoerni, and everybody else. “If you come late,” Noyce liked to say, “you just have to park in the back forty.” And there would be no baronial office suites. The glorified warehouse on Charleston Road was divided into work bays and a couple of rows of cramped office cubicles. The cubicles were never improved. The decor remained Glorified Warehouse, and the doors were always open.

I suspect this happens a lot to scientists and engineers too. Almost every experience they have is analyzed. Ever been to an arena rock concert and wondered about the stage setup, the sound, the lights, the speakers, the pickups, the mics? Read something that wants to move your soul but constantly wondered about the writer and his process? Then, you too, have the curse of analysis.

There is now a vast body of research that that documents broad behaviors that make us happy, as well as the cognitive biases that trick us into straying from that path. Yes, happiness, that fuzzy touchy-feely concept is rapidy becoming an empirical science. And that’s great!

I recently came across several papers that drove this point home for me. It is well known that money does indeed buy happiness, but only up to a point. But why does happiness plateau after a certain level of material fulfillment? How can you better use your money to increase your happiness?

Robert Frank, in “How Not to Buy Happiness”¹ talks about how channeling money away from conspicuous consumption will make you happier. He constructs two imaginary universes, one in which everyone has huge houses, but no free time for exercise, very little time to spend with friends, and very little vacation, and another one where the houses are just a little smaller (3000 sq ft vs 4000 sq ft), but where people time for exercise and friends and vacations. Study after study has shown that we will be happier in the second universe. And yet, most developed nations are steadily moving towards the first. Frank posits that this is because everyone else is buying larger and larger houses, which makes the one with the small house feel inadequate. It’s an arms race.

In “If money doesn’t make you happy, then you probably aren’t spending it right”², the authors start with the premise:

If money can buy happiness, then why doesn’t it? Because people don’t spend it right. Most people don’t know the basic scientific facts about happiness—about what brings it and what sustains it—and so they don’t know how to use their money to acquire it. It is not surprising when wealthy people who know nothing about wine end up with cellars that aren’t that much better stocked than their neighbors’, and it should not be surprising when wealthy people who know nothing about happiness end up with lives that aren’t that much happier than anyone else’s. Money is an opportunity for happiness, but it is an opportunity that people routinely squander because the things they think will make them happy often don’t.

They go on to explain all the cognitive biases and adaptations that routinely push us towards the choices that will ultimately bring us unhappiness. For example, our capacity for rapid adaptation means that shizy gizmo’s glean wears off quickly, leaving us mystified why we don’t feel as good as we thought we would when we lusted after it. They prescribe eight concrete behaviors which research has shown will likely increase your happiness. Go read the paper – it is a quick, pleasant read, worth every minute. Among their recommendations: buy experiences rather than things, and buy many small pleasures instead of a few big ones.

But by far the most surprising study was one that finds the causal link between unhappiness and a wandering mind³. If you ask almost anyone, they will say that unhappiness causes the mind to wander. If life sucks, you look around for anything else other than your current situation. But the authors prove the opposite. Unhappiness is caused by a wandering mind. In other words, being focused will make you happier. Don’t try to be happy first so that you can then be focused. (They used a neat time-delay analysis to tease apart the causal link.)

Will we pay heed?

Update, June 26, 2012: I recently came across another interesting paper in this vein: If Money Doesn’t Make You Happy, Consider Time⁴. It talks about how we should spend our time to increase our happiness.

This impending “flash crisis” is well known in system circles. It’s almost like a mini Y2K.

However, its repercussions are much deeper than just the choice of programming language and runtime³.

It’s depressing to see graphs of things like CPU speed and disk and RAM capacity, which keep going up, up, up, compared with those of disk throughput and seeks, which for all practical purposes has remained mostly flat. A 100 GB drive from a few years ago does about the same number of seeks per second as a 2 TB drive today. Drive seeks have become the straw through which we’re trying to drink the ocean.

If you look at the current software stack, all the way down to the operating system kernel, you will find that the slowness of disks is a baked-in assumption throughout. Operating systems play intricate games with IO scheduling and how files are laid out on disk and caching disk blocks in memory, and spend millions of CPU cycles doing so, because the few microseconds taken for those millions of CPU cycles is noise compared to the tens of milliseconds it might take to hit the disk. Pushed to the extreme, this obsession with avoiding the disk leads to schemes like RAMSter⁴, because even accessing the memory of a different computer over the network fabric is faster than going to local disk.

In essence, we have been trading plentiful resources such as CPU cycles and RAM for precious disk seeks. Most web-ish serving workloads almost demand this, because they tend to be IO rather than CPU bound.

What happens when you blow up the “slow disk” assumption is that your entire software stack begins to crumble. When the cushion provided by those 10 ms of seek time goes away, CPU becomes your bottleneck. You have to start rethinking a lot of design choices.

There are a few saving graces which will buy us time. Flash hasn’t hit the price point where it can en masse replace spinning disks⁵. And the shakedown will start from the top of the stack. Programmers will first discover bottlenecks in their application code, then in their choice of language and runtime, and finally, after those layers have been tuned to death will they look at things like filesystems and the operating system kernel.