Blog

A Decade as a Linux Pro

After I recently accepted my old age, I was talking to some friends who, to their own surprise, found they were old too. We were talking about when each of us started working with Linux and a friend noticed he had been working with Linux for 10 years. That’s when I realized it’s been a decade since I was first paid to work with the operating system.

I had been using Linux for a little while. My first contact with it was in mid 1995. I had bought a computer magazine from the UK that had this pink CD-ROM with something called Linux-FT.

One of the cool things about Linux-FT was that it had a licensed copy of the Motif window manager, which was pretty cool at the time. I had been using some RISC boxes in college running CDE, so I felt right at home.

infomagic
Not long after that, a friend who was an administrator at a new ISP let me borrow his Walnut Creek CD-set containing Slackware 2.1 or 2.2 (I’m not sure anymore. Old age, remember?) Then I purchased a copy of the wonderful InfoMagic 5-CD set containing Slackware, Red Hat, mirrors of some FTP site and another distro I can’t remember either (Shit! I’m old…)

Then in 1999 I was hired to help this company migrate from Windows to Linux. It was the first time I was ever paid to do anything related to Linux.

I went to work at Conectiva the next year, where I learned I didn’t know anything. What the heck do I do now? That’s also where I learned like never before, made lots of friends, and found the love of my life. (No, not another Linux distribution! I mean my wife! What’s wrong with you?)

In 2004 I left Conectiva and started working with one of the company’s founders on another Linux project. The work itself was interesting, but it was also my first contact with something that I would see a lot more in the future: the downright dishonesty of Linux entrepreneurs in Brazil. Of course, I then thought it was an isolated thing and decided I didn’t want to be a part of that and left the company in 2005.

That’s when I came to Intel to work at the CSO, a “personal” project of Andy Grove. CSO had a simple mission: to foster Linux usage by financing and providing engineering to business with good ideas. How great is that? I was going to work on my passion (Linux) and meet all kinds of people who shared it with me. What could possibly go wrong?

Oh boy.

In the following year I’d see things that would still make me sick years later. From businesspeople to self-appointed free software leaders, all I saw was guile and greed. It was such a disappointment that I requested a change. I stopped working with Brazilian businesses and projects completely, moving to support projects in other countries. Things were much better, which is another disappointment and one of the reasons why I hold us Brazilians in such low regard as a people.

Outside Brazil things were very different and despite some funny things here and there, I am proud of the Linux work I’ve done, especially the megalarge project with the government of Venezuela. I even met Hugo Chavez, which is funny considering my political stand 🙂

Regardless, I was also getting disappointed at other things as well. After years developing projects such as KDE, I was bored to death. I started witnessing things being done on other platforms and suddenly the Linux desktop just felt stale to me. It was dull and lifeless and at the same time I was doing all these cool stuff on Windows.

Add to all that the fact that KDE started getting full of kids adding more and more useless features… and politics… ever heard the one about the moppet who decided that all KDE About boxes should contain a thank-you note to American troops worldwide? And he wasn’t even American? You know what?

Screw Linux.

I dropped it like you wouldn’t believe. And it felt good. Not having to edit configuration files to do something simple made stuff pleasant again. I was introduced to Mac OS X and I loved it. It was Linux on steroids. All the good stuff without the kludge.

No more politics. No more GNU Slash Linux. No more open source vs. free software. No revolutionary-audio-framework-of-the-month. No juvenile cockiness. No rWindoze. No Micro$oft.

Just fun.

After nearly 15 years, I’m truly free.

The Grumpiest Post I’ve Ever Written

So today I was called grumpy, which was inconsiderate and uncalled-for, so I did what every stable adult man would do. I complained to the one person who’s required by God and Country to always be supportive: my wife. “Who the hell does she think she is calling me grumpy?” I asked. And my wife went “well, you totally are,” which was an inconsiderate and uncalled-for answer. “Hey! You’re my wife! You’re supposed to love me!” and she was “I do love you! And that’s why I need to be honest.”

Crap.

It’s not that I could not live with the idea of being grumpy. It’s just that I don’t see myself like that. Or rather I didn’t. Now I kind of do.

There are a couple of young kids who are all high on the whole GNU Fucking Slash Linux. Not unusual. When we’re young, the world does seem a lot more black and white. That’s fine. Kids will rebel and Linux… Ops, sorry! GNU Slash Linux provides an outlet from which all the rebellion that a young nerd can muster. I mean, it’s not like these kids get invited to the parties where popular kids do their rebelling, right? Again, that’s all fine and dandy. Been there.

But then these kids—high as they are on GNU Slash Kool-aid—forget that they’re the new kids on the block. I mean, seriously people, Linux isn’t new anymore. There is a lot of people who came before you kids came out of daddy’s place. No shit.

Let’s face it, this is part of being young. You feel like you have all the answers and everyone else in the world failed to see The Truth. No problem with that. Except…

Except that shit means I’m now officially old!

Motherfucker.

Today I see a friend commenting about Linux and one of these cocky, know-it-all kids jumped on his case like he was some kind of newbie or something. Hell, that actually got me angry. And then it hit me.

Doesn’t that make me grumpy?

So. Fucking. Be. It. I hereby accept the label of grumpy and shall sport it with pride.

Grumpiness

So my friend works with and improves Linux for years and then a kid comes out of nowhere—having never done more than tweeting about Linux—and dares berate the guy? Go suck an egg, son! The guy I’m talking about knows more Linux—and note how I dropped that slash shit and I don’t even care—on his pinky toe than all your friends together. Kid!

Goddammit.

-rst.

P.S.: I’m not really angry. Or am I?

The Delusion of Not Being Deluded

Ever came across someone who wanted to show off how independent and free their own thinking was? I have, many times. Discussing with these people is an amusing and yet wretched event. It’s like goping to a funny dentist. She may be funny, but she’s still going to bore a deep hole in your tooth. And boy, will it hurt! At least one of you will be laughing, right?

I happen to know a couple of people who have got such strong conviction of their own uniqueness in being free from delusion that they cannot avoid being delusional themselves. Both are on this righteous crusade to extricate people from their self-foisted enslavement in the hands of evil Apple. Wait, what? Yes, that’s right. From Apple! Apparently to some people, the operating system or the phone we use is the wrong one and we are pulling the wools over our own eyes into thinking otherwise. Fortunately for all of us, there are these cavaliers in shiny armor who will rescue us from ourselves. Dude, seriously? How much more fucked-up delusional can one get?!

image

I don’t know if those guys did too much role-playing in their lives (and I don’t mean in bed) or it’s just too much World of Warcraft. I don’t really know. But if there is one thing I do know, it’s that it’s delusional. Why is this so? It’s really a synthesis of a couple of things —

  • Belief that one is right about something. Being fair, we are all like this. If you ponder about it, this is the very definition of having an opinion. Nobody thinks they are wrong about their opinions.

  • Self-aggrandizement. Again, to a variable degree, we’re all like that. We all love to think we are more than what we really are. Probably a defense mechanism. Without that, we’d all have killed ourselves by the time we reached puberty. Or maybe not all of us, because I am super awesome.

  • Strong illusions.

What exactly is a delusion? A delusion is the belief that an illusion is real. We all have illusions, but a few go farther. They have such strong illusions that they begin believing in them.

These people actually believe that there is a war going on. A struggle between good and evil. A conflict between freedom and slavery. And let’s face it, everyone wants to be one of the good guys. Except supervilains. Supervilains want to be on the wrong side. Always.

Once you are set on the idea that (a) there is something like black-and-white good and evil, and that (b) the fucking clash is on, there’s really no stopping you. You need to fight to Good Fight. But since there really isn’t a war going on, the only way to fight it is by making up stuff as you go. So you pick your villains (say, Microsoft and Apple) and your heroes (say, Google and GNU) and you go out saving us ignorant civilians. From ourselves. And from Apple. Because Apple is, like, super evil. Ah, if everyone was like Google…

Holy. Shit. People. Grow the fuck up!

Let’s all pretend for a moment that there really is a war going on. That Google is out there fighting for our rights to free goodies in exchange for nothing at all. That Microsoft really owns the world’s Secret Cabal and Bill Gates sits down everyday to plan what evil deed needs to be done that day. Let’s say all of this is real. Wouldn’t it be a lot more productive for you to focus on your enemies and leave us civilians alone? Go ahead, wear your fancy penguin costume and take pictures of yourself peeing on the Microsoft logo. A lot of people will love that. And more importantly, you will feel like you’ve finally accomplished something! You won a battle for the good guys! Hooray!

But please leave the rest of us alone.

-rst.

P.S.: a friend just sent me this picture, which says a lot. I think the message is: Dude, don’t be a fag!

image

perf Turns Counters Into Questions

I had two versions of a parser. One finished faster, so I declared victory and nearly deleted the slower one. Before doing that, I tried the new perf tools included with recent kernels and discovered that my explanation for the improvement was wrong.

The kernel’s performance-counter infrastructure provides a common way to measure hardware and software events. The perf tool can count events for a command, sample execution and report where those samples landed. Instead of beginning with a profiler tied to one processor model, I can ask through one kernel interface and use the events available on this machine.

perf stat is a useful first question. It reports elapsed time along with counts such as cycles, instructions, context switches and page faults. Ratios matter more than isolated totals. Instructions per cycle can suggest whether the processor is retiring useful work efficiently, while cache-related events may explain why an apparently smaller algorithm still stalls.

My faster parser did not execute dramatically fewer instructions. It incurred fewer cache misses because its data was laid out more compactly. I had credited a clever branch change that happened nearby. The benchmark result was real; my story about it was fan fiction.

Sampling answers a different question. perf record periodically captures the current instruction pointer, and perf report aggregates samples by symbol. With suitable symbols, hot functions become visible without instrumenting every call. Sampling has overhead and statistical uncertainty, but it is often much less disruptive than logging entry and exit around a hot path.

Counters are constrained resources. The processor can measure only a limited number simultaneously, so events may be multiplexed. Some events are model-specific, and virtualized or restricted environments may expose less. A cache-miss count without knowing which cache or how the event is defined is an attractive number with an uncertain biography.

I also avoid optimizing from one profile. Workload, input size, compiler options and machine state all matter. I run repeated measurements, preserve the input and compare complete behavior. A ten-percent gain in a microbenchmark is not useful if the changed layout doubles memory for the real service.

I strongly prefer starting performance work with perf stat, then sampling when the totals suggest a question. ftrace remains better for many scheduling and kernel-flow investigations; perf is especially convenient for connecting program hotspots to processor and kernel counters. Neither tool replaces understanding, but both are considerably more reliable than staring at source until one loop begins to look guilty.

Go Is an Interesting Experiment

Google announced a new language called Go last week, and I spent an evening rewriting a small concurrent program in it. This is not a review. Go is new and experimental, its tools are young, and its APIs are changing. Anything written today may become an historical document by next month.

The appealing part is its systems-shaped simplicity. It has compiled code, pointers and familiar control structures, but also garbage collection, interfaces, goroutines and channels. Starting concurrent work is cheap in syntax, while channels provide a way to pass values instead of sharing every structure behind a lock.

My first version translated threads directly into goroutines and retained all the old shared state. Unsurprisingly, new spelling did not improve the design. The better version gave one goroutine ownership of the state and sent it requests through channels. The useful idea was ownership, not the keyword.

There are rough edges, incomplete libraries and unanswered questions about performance and deployment. I would not build an important product around today’s interfaces. I would use it for experiments, particularly network services where its concurrency model can be exercised honestly.

I strongly like the direction, with emphasis on direction. Go is not yet a settled platform, and confidence would be silly after one week. Still, it has made me think differently about a program I already understood, which is a fine result for an evening and cheaper than another programming book.