Linux

Goroutines Are Not Pthreads

My reflex when seeing go f() was to translate it into “start a thread.” That translation is convenient and wrong enough to cause bad designs.

A goroutine is a language-level activity scheduled by the Go runtime. Its initial stack is small and can grow, so creating thousands is practical in cases where creating thousands of pthreads would be a conversation with both the kernel and the person carrying the pager.

I tested this with a deliberately dull program:

package main

import "fmt"

func wait(done chan bool) {
    done <- true
}

func main() {
    done := make(chan bool)
    for i := 0; i < 10000; i++ {
        go wait(done)
    }
    for i := 0; i < 10000; i++ {
        <-done
    }
    fmt.Println("finished")
}

The channel is doing two jobs: carrying a value and synchronizing sender with receiver. With an unbuffered channel, the send waits for a receive. The program does not need a mutex around a shared completion counter.

That does not make races impossible. If all those goroutines update the same map, I still have a race. “Do not communicate by sharing memory” is good direction, not a force field.

The early runtime is another caveat. On the snapshot I used, scheduling and stack management are active work. Timing results vary, and a tight goroutine that never calls into the runtime can be an unfriendly neighbor. I would not build a latency promise from this experiment.

The cheap creation cost also does not excuse leaving goroutines blocked forever. Their stacks are small, not imaginary, and their work still needs a defined end.

What I do like is the change in scale. In C, a thread is sufficiently costly and awkward that I first ask how to avoid one. In Go, I can model each independent operation directly, then decide where synchronization belongs. The runtime may multiplex many goroutines over fewer operating-system threads; that implementation is precisely why “goroutine equals pthread” is a poor mental model.

This is not magic parallelism either. More runnable work does not create more processors, and today’s defaults may use only one. It does make concurrency cheap enough to express before I optimize it. For server and pipeline code, that changes the first sketch substantially.

A Decade as a Linux Pro

After I recently accepted my old age, I was talking to some friends who, to their own surprise, found they were old too. We were talking about when each of us started working with Linux and a friend noticed he had been working with Linux for 10 years. That’s when I realized it’s been a decade since I was first paid to work with the operating system.

I had been using Linux for a little while. My first contact with it was in mid 1995. I had bought a computer magazine from the UK that had this pink CD-ROM with something called Linux-FT.

One of the cool things about Linux-FT was that it had a licensed copy of the Motif window manager, which was pretty cool at the time. I had been using some RISC boxes in college running CDE, so I felt right at home.

infomagic
Not long after that, a friend who was an administrator at a new ISP let me borrow his Walnut Creek CD-set containing Slackware 2.1 or 2.2 (I’m not sure anymore. Old age, remember?) Then I purchased a copy of the wonderful InfoMagic 5-CD set containing Slackware, Red Hat, mirrors of some FTP site and another distro I can’t remember either (Shit! I’m old…)

Then in 1999 I was hired to help this company migrate from Windows to Linux. It was the first time I was ever paid to do anything related to Linux.

I went to work at Conectiva the next year, where I learned I didn’t know anything. What the heck do I do now? That’s also where I learned like never before, made lots of friends, and found the love of my life. (No, not another Linux distribution! I mean my wife! What’s wrong with you?)

In 2004 I left Conectiva and started working with one of the company’s founders on another Linux project. The work itself was interesting, but it was also my first contact with something that I would see a lot more in the future: the downright dishonesty of Linux entrepreneurs in Brazil. Of course, I then thought it was an isolated thing and decided I didn’t want to be a part of that and left the company in 2005.

That’s when I came to Intel to work at the CSO, a “personal” project of Andy Grove. CSO had a simple mission: to foster Linux usage by financing and providing engineering to business with good ideas. How great is that? I was going to work on my passion (Linux) and meet all kinds of people who shared it with me. What could possibly go wrong?

Oh boy.

In the following year I’d see things that would still make me sick years later. From businesspeople to self-appointed free software leaders, all I saw was guile and greed. It was such a disappointment that I requested a change. I stopped working with Brazilian businesses and projects completely, moving to support projects in other countries. Things were much better, which is another disappointment and one of the reasons why I hold us Brazilians in such low regard as a people.

Outside Brazil things were very different and despite some funny things here and there, I am proud of the Linux work I’ve done, especially the megalarge project with the government of Venezuela. I even met Hugo Chavez, which is funny considering my political stand 🙂

Regardless, I was also getting disappointed at other things as well. After years developing projects such as KDE, I was bored to death. I started witnessing things being done on other platforms and suddenly the Linux desktop just felt stale to me. It was dull and lifeless and at the same time I was doing all these cool stuff on Windows.

Add to all that the fact that KDE started getting full of kids adding more and more useless features… and politics… ever heard the one about the moppet who decided that all KDE About boxes should contain a thank-you note to American troops worldwide? And he wasn’t even American? You know what?

Screw Linux.

I dropped it like you wouldn’t believe. And it felt good. Not having to edit configuration files to do something simple made stuff pleasant again. I was introduced to Mac OS X and I loved it. It was Linux on steroids. All the good stuff without the kludge.

No more politics. No more GNU Slash Linux. No more open source vs. free software. No revolutionary-audio-framework-of-the-month. No juvenile cockiness. No rWindoze. No Micro$oft.

Just fun.

After nearly 15 years, I’m truly free.

perf Turns Counters Into Questions

I had two versions of a parser. One finished faster, so I declared victory and nearly deleted the slower one. Before doing that, I tried the new perf tools included with recent kernels and discovered that my explanation for the improvement was wrong.

The kernel’s performance-counter infrastructure provides a common way to measure hardware and software events. The perf tool can count events for a command, sample execution and report where those samples landed. Instead of beginning with a profiler tied to one processor model, I can ask through one kernel interface and use the events available on this machine.

perf stat is a useful first question. It reports elapsed time along with counts such as cycles, instructions, context switches and page faults. Ratios matter more than isolated totals. Instructions per cycle can suggest whether the processor is retiring useful work efficiently, while cache-related events may explain why an apparently smaller algorithm still stalls.

My faster parser did not execute dramatically fewer instructions. It incurred fewer cache misses because its data was laid out more compactly. I had credited a clever branch change that happened nearby. The benchmark result was real; my story about it was fan fiction.

Sampling answers a different question. perf record periodically captures the current instruction pointer, and perf report aggregates samples by symbol. With suitable symbols, hot functions become visible without instrumenting every call. Sampling has overhead and statistical uncertainty, but it is often much less disruptive than logging entry and exit around a hot path.

Counters are constrained resources. The processor can measure only a limited number simultaneously, so events may be multiplexed. Some events are model-specific, and virtualized or restricted environments may expose less. A cache-miss count without knowing which cache or how the event is defined is an attractive number with an uncertain biography.

I also avoid optimizing from one profile. Workload, input size, compiler options and machine state all matter. I run repeated measurements, preserve the input and compare complete behavior. A ten-percent gain in a microbenchmark is not useful if the changed layout doubles memory for the real service.

I strongly prefer starting performance work with perf stat, then sampling when the totals suggest a question. ftrace remains better for many scheduling and kernel-flow investigations; perf is especially convenient for connecting program hotspots to processor and kernel counters. Neither tool replaces understanding, but both are considerably more reliable than staring at source until one loop begins to look guilty.

Btrfs Is Interesting, Not Yet Boring

Btrfs entered the mainline kernel earlier this year, so naturally I created a filesystem and started putting unimportant data on it. Filesystems are fascinating, provided the nouns “only copy” and “family photographs” remain elsewhere in the sentence.

The central structure is a copy-on-write B-tree. When metadata changes, Btrfs writes new blocks rather than overwriting the existing path in place. Data can also be copy-on-write. This makes snapshots and clones natural consequences of the storage model instead of layers awkwardly attached above a filesystem.

Snapshots are cheap at creation because they initially share existing extents. As either side changes, new extents represent the differences. That is attractive for test trees: I can capture a known state, run an unpleasant upgrade and return without copying every unchanged file. Shared blocks also mean deletion and free-space accounting are less obvious than in a simple independent copy.

Checksums cover data and metadata, allowing corruption to be detected rather than quietly delivered. With redundant storage, the design can potentially use a valid copy to repair damaged data. Detection alone remains valuable, although it cannot reconstruct bytes that exist in only one damaged place. A checksum is an alarm, not a backup wearing mathematical spectacles.

Btrfs also aims to manage multiple devices and provide subvolumes, online growth, compression and other facilities usually assembled from several layers. Integrating them allows the filesystem to understand which blocks contain live data, which can help operations that would otherwise see only an undifferentiated block device.

The ambitious scope is exactly why I am cautious. Mainline inclusion makes development and testing easier; it does not make the format or implementation mature overnight. Repair tools, failure behavior and operational knowledge matter more than a successful benchmark. A filesystem earns trust through years of dull recoveries, not a feature table.

My current test uses build outputs and source trees already stored elsewhere. I exercise snapshots, fill the filesystem, delete subvolumes and simulate untidy shutdowns. Performance is interesting, but I care more about whether every operation has understandable results and whether tools can explain the state afterward.

I prefer Btrfs’s integrated copy-on-write direction for experiments that benefit from snapshots. I do not yet prefer it for important production data. ext4 is the conservative choice today, and even that choice requires backups. Btrfs is early and explicitly experimental; the most useful contribution ordinary testers can make is finding failures without turning those failures into personal disasters.

KMS Moves the Screen Into the Kernel

My test laptop changes display modes several times while booting: firmware text, a console mode, the X server’s preferred resolution, and sometimes another mode after resume. Each transition blanks the screen long enough to make me wonder whether the machine is considering a different profession.

Kernel mode setting moves responsibility for configuring display modes into the kernel graphics driver. Traditionally the X server performed that setup from user space. With KMS, the kernel knows about connectors, CRTCs, encoders, framebuffers and modes early enough to establish the display before X starts.

That early ownership makes boot transitions smoother, but the architectural benefit is larger. Suspend and resume already require kernel coordination with devices. If display state is also managed there, restoration does not depend on a particular user-space server reconstructing everything after the fact. Panic and virtual-terminal paths can retain a usable framebuffer as well.

The Direct Rendering Manager interface exposes the resources to user space. A display server can discover connectors and available modes, choose how framebuffers map to scanout hardware, and request changes through a common kernel boundary. The kernel arbitrates access to the device rather than allowing unrelated programs to reprogram it optimistically.

KMS also fits the broader memory-management work in graphics drivers. Rendering buffers and scanout buffers share hardware constraints, and transitions between them need synchronization. Putting mode setting beside DRM’s device and buffer management reduces the number of components that must agree through private arrangements.

My Intel machine with a recent kernel now reaches its native mode earlier and switches to X with less flashing. Resume has also become more predictable. This is not universal evidence. Driver support varies, hardware contains creative surprises, and a failed early mode set can replace a merely ugly boot with a completely dark one.

Debugging changes too. Problems that once belonged entirely to the X log may now begin in kernel messages and driver parameters. Distributions need rescue paths and conservative defaults while support matures. The mechanism is better placed, but moving code into the kernel also raises the cost of mistakes.

I strongly prefer KMS as the long-term division of responsibility. Display hardware is a shared device whose state matters before, during and after the graphical session; the kernel is the natural owner. For current machines I enable it where the driver is known to work and keep a boot option that disables it. Architectural conviction is comforting, but a visible console is more immediately useful.