Blog

I rewrote a small Python log processor as three stages: read lines, parse records, and aggregate counters. The Python version used queues and worker threads. The Go version uses channels, mostly because I wanted to discover where they stop being cute.

The parser looks like this in the weekly snapshot I have installed:

func parse(in <-chan string, out chan<- Record) {
    for line := range in {
        r, ok := parseLine(line)
        if ok {
            out <- r
        }
    }
    close(out)
}

Directional channel types document who sends and who receives. They also let the compiler reject an accidental send in a consumer. The reader closes its output, the parser drains that input and closes the next channel, and the aggregator stops when its input is closed.

An unbuffered channel forces a rendezvous. If aggregation falls behind, parsing blocks, then reading blocks. That is back pressure without a separate queue-length policy. For this tool it is desirable because there is no value in reading a gigabyte ahead merely to admire it in memory.

A buffered channel changes the timing, not the ownership:

records := make(chan Record, 64)

Now the parser can get 64 records ahead. I tried capacities from zero to 4096. Small buffers smoothed disk and parser timing; large ones mostly increased memory use. The best number depended on record size and machine, so naturally 64 is now a sacred constant until the next measurement.

Closing requires discipline. The sender should close a channel when no more values will arrive. A receiver closing it is guessing about other senders and can provoke a panic. Multiple producers therefore need one coordinator to wait for them and close the shared output.

The current implementation is experimental, as are some channel syntax and library details around it. This is not a claim that channels beat every queue or lock. A shared read-mostly table may be clearer behind a mutex. But for a pipeline, channels put flow control, synchronization, and the handoff of values in one visible operation. I removed two condition variables and, more importantly, stopped having to prove that I signaled both of them on every path.

I usually notice a missing cleanup path during review, which is a dignified way of saying after I wrote it.

Go’s defer lets me put cleanup beside acquisition:

f, err := os.Open(name, os.O_RDONLY, 0)
if err != nil {
    return err
}
defer f.Close()

The call runs when the surrounding function returns, including an early return. Several deferred calls run in last-in, first-out order. That order fits nested resources rather nicely.

Arguments are evaluated when the defer statement executes, not later. This matters in loops and when variables change:

for i := 0; i < 3; i++ {
    defer fmt.Println(i)
}

The output is 2, 1, 0. It is a stack of already-prepared calls, not three promises to inspect i at the end.

I would not put defer in a hot inner loop without measuring it. This compiler and runtime are young, and convenience has a cost even when the syntax hides the invoice. A helper function can give each iteration its own cleanup boundary.

The larger win is boring correctness. Files close on every return, locks can be paired with unlocks, and the cleanup is visible before the function accumulates three branches. C’s goto out remains perfectly respectable, but defer makes the common case harder to omit. This note matches my current weekly snapshot; the surrounding library calls are still moving.

I’m not a big believer in keeping books forever, even though I do keep some. Most books, however, I just want to read and pass along.

It just so happens that I have a ton of books that I want to get rid of. I could throw them away, but it somehow feels wrong. Also, I could donate them to some school, but considering where I live now, I assume this too will bring me one hell of a bureaucratic nightmare. Also, most of the books are either in English or Portuguese, so schools would probably not use them anyway.

With very few exceptions, all the books are either mathematics, economics, programming, or scifi. I thought I’d simply take the books to the office and give them away to anyone who’d like to have them.

But then my wife gave me an idea. Instead of taking the books and giving them outright, I started sending random emails to a mailing list with geeky questions, often related to the book topic. Whoever answers correctly first, gets the book.

This worked amazingly well.

1. It created a fun environment for all involved. People actually want to get the answers right. Others learn by, well, learning the winning answer.

2. For books that normally no one would care about, the fact that you have to win to get it suddenly adds value. It’s no longer a book no one wants, it now is an award.

3. All of a sudden everybody loves me 😉

Had I just announced I had a bunch of books, people would get by, pick a few they cared about and that would have been it. No fun. So every now and then I take a few books from home and then give them away to quiz winners.

It’s actually lots of fun.

Coincidences have a way of catching one’s attention, haven’t they? Yesterday I was riding the bus home with a colleague from work. I was telling him how my life here wasn’t perfect because I couldn’t buy a car. It turns out that I would need to hold a national ID card (called DNI) to purchase a car. Actually, one needs to obtain the DNI to do pretty much anything like opening a bank account. Interestingly enough, you need the DNI for small, simple tasks like ordering cable TV, phone lines, Internet access, etc, but one can easily buy a house – or many houses – without the DNI.

The process of obtaining the DNI is simple. If you’re a citizen, your parents will have used your birth certificate to get your first DNI, which is mandatory from a very early age. Over the years, you are required to update your DNI when you reach certain ages (8, 16, 25, if I’m not mistaken) and everytime you change address. All you need is the original DNI and some address verification and you’re done. If you’re an expat like myself, you need your passport, your resident visa, and some address verification. Again, very simple. You go, apply and wait for the document to be ready. And that’s where the problem resides… in parts of the country, the DNI takes years to be ready. Even in Buenos Aires, it can take more than one year for your DNI to be ready for pick-up.

Now how could such a simple and yet much needed document take so long to be manufactured?

So there I was, talking to this colleague about all of this. He told me that when he had to change his at 16, it took two years. And he’s a national. He told me it’s usually because government workers don’t have the incentive to work harder. He’s probably right, as this is the case everywhere else in the world. Still, this might explain why a simple document can take 2-3 weeks to be created, but not years. When I stepped out of my ride, I bumped into a girl who lives in our building. She told me she was frustrated because she had lost her DNI a couple of years back and the new one wasn’t ready yet. She had just went to the Identification Office to check. Coincidences… I then told her about my story and she told me so other tales of horror and I shared some of my own, like when I needed a document notarized and it took a month to do it. And that was it.

Fast forward a day.

My wife is currently having classes and she asked me to get her a textbook she needs. I went to this bookstore on my way to work. I strolled into the store and worked my way to the textbooks section and quickly found the one my wife needed. I picked it up and headed to the checkout, content that I wouldn’t have to hunt for the book in other stores. I held the book and the cash in my hand. “I’d like to take this” I told the lady at checkout, who in turn looks around and goes “Did you pick a number?”

What. Fucking. Number.

“What number?” I asked, confused. She then pointed me to this little thingamajig near the entrance. “I’m sorry, I don’t get it…” And then she half-mockingly explained to me that I had to pick the number and wait my turn for a salesperson to help me. “But I don’t need any help. I have the book and the money right here!”

No use. I had to pick a number and wait for one of the two saleswomen to call it. So there I was, holding the book I wanted, with the cash to pay for it, standing by the checkout – where the lady was doing nothing by the way – and waiting for someone to “help me.” In any half-civilized place our business would have been over by now. The store would have made money and I would be on my way to work. Simple.

“Twenty!!!!!!!” shouted a lady. It was my number. So I went to her – looking pissed, I’m sure. I told her I’d like to please take this textbook. “Oh, certainly!” So she goes to a computer terminal and asks if I’m paying with a credit card. I wave the cash and say “efectivo” to make sure she gets it. Then she asks what the book is for. For reading, duh! I tell her that it’s for a course, then she types something and asks which school. “Lady, sorry but all I want to to take this textbook with me. I have the cash right here. I’ve had both the book and the cash in my hands for like 15 minutes already. All that’s missing is for that lady at checkout to get the money so I can get out of here.” And yes, I was pissed and I didn’t really sounded friendly and half the store was staring at me. She said apologetically, “I’m sorry, but I need to register who purchased the book and where it will be used.”

Why?!?!

“Lady, seriously, I don’t want to give any more information. I want to buy this book and that’s it. You either take my money or not. Your choice. Just let me know so I can get out of here.”

The whole store was following our exchange. Everyone surely thinking that I was the mean foreigner who’s going to explore them and steal their water (it’s a local thing.) She told me to go to the checkout I had been to 15 minutes earlier. The lady took my money and asked if I wanted a receipt. D-U-H!

So I realized it has to be a cultural thing. Bureaucracy is so entrenched in their lives that they don’t even realize it anymore. They just think it’s the best way of doing things. The problem I had to get a document notarized is a perfect example. I went to a notary, who then notarized my document. It took two days, which is already ridiculous, but still… the problem is that after he notarizes it, he has to send the document to the Notary Guild Office so that they in turn can verify that the notarizing notary was in fact a certified notary and that the notarization was correctly notarized. I am not even kidding.

Argentina has a lot of great things. Things that I actually love. But in terms of bureaucracy, it’s really, really a mess. And the very people have been “infected” by it so that they are bureaucratic at heart. It’s actually really sad.

My reflex when seeing go f() was to translate it into “start a thread.” That translation is convenient and wrong enough to cause bad designs.

A goroutine is a language-level activity scheduled by the Go runtime. Its initial stack is small and can grow, so creating thousands is practical in cases where creating thousands of pthreads would be a conversation with both the kernel and the person carrying the pager.

I tested this with a deliberately dull program:

package main

import "fmt"

func wait(done chan bool) {
    done <- true
}

func main() {
    done := make(chan bool)
    for i := 0; i < 10000; i++ {
        go wait(done)
    }
    for i := 0; i < 10000; i++ {
        <-done
    }
    fmt.Println("finished")
}

The channel is doing two jobs: carrying a value and synchronizing sender with receiver. With an unbuffered channel, the send waits for a receive. The program does not need a mutex around a shared completion counter.

That does not make races impossible. If all those goroutines update the same map, I still have a race. “Do not communicate by sharing memory” is good direction, not a force field.

The early runtime is another caveat. On the snapshot I used, scheduling and stack management are active work. Timing results vary, and a tight goroutine that never calls into the runtime can be an unfriendly neighbor. I would not build a latency promise from this experiment.

The cheap creation cost also does not excuse leaving goroutines blocked forever. Their stacks are small, not imaginary, and their work still needs a defined end.

What I do like is the change in scale. In C, a thread is sufficiently costly and awkward that I first ask how to avoid one. In Go, I can model each independent operation directly, then decide where synchronization belongs. The runtime may multiplex many goroutines over fewer operating-system threads; that implementation is precisely why “goroutine equals pthread” is a poor mental model.

This is not magic parallelism either. More runnable work does not create more processors, and today’s defaults may use only one. It does make concurrency cheap enough to express before I optimize it. For server and pipeline code, that changes the first sketch substantially.