Blog

I wrote a small fan-out test and expected twice as many goroutines to mean twice as much parallel work. The result barely moved. That sent me away from the programme and into the runtime scheduler.

The test looked innocent:

for i := 0; i < 4; i++ {
	go crunch(work[i])
}

Goroutines are cheap concurrent activities, but concurrency does not itself promise simultaneous execution on every processor. In the current runtime, scheduling is still comparatively simple, and processor use depends on runtime settings, operating-system threads, blocking calls, and where the scheduler gets a chance to run.

The surprise was how many layers my little loop depended upon. A goroutine begins with a small stack and runtime bookkeeping. Runnable goroutines wait for the scheduler. The scheduler places work onto operating-system threads, and the kernel finally decides when those threads run. Linux 3.0 may schedule the threads beautifully, but it cannot create parallel Go execution the runtime never offered it.

This scheduler predates the more ambitious designs being discussed for later Go. It should not be described using machinery that does not exist yet. There is no reason to draw a modern diagram over an old runtime and then congratulate the diagram for explaining it.

Blocking is the interesting case. If a goroutine enters a system call, the runtime may need another thread so runnable Go code can continue. Channel operations and synchronization also move goroutines between runnable and waiting states. The scheduler therefore coordinates cheap user-level goroutines with more expensive kernel threads rather than mapping one permanently to the other.

I prefer to write synchronization for correctness first and treat parallel speedup as a measured property. A channel can make ownership clear even on one processor. If CPU parallelism matters, I set the runtime’s processor allowance explicitly, run a long enough workload, and compare against a serial version.

The caveat is that scheduler behaviour is an implementation detail in active development. A trick that coaxes today’s scheduler may become useless or harmful after a snapshot. Explicit communication and coarse useful work are safer investments than depending on an accidental scheduling order.

My test eventually did improve, but only after I made each unit of work large enough to outweigh coordination and enabled the processors I intended to use. Four goroutines were not four tiny employees waiting inside the computer. This was disappointing for payroll, but helpful for the benchmark.

Conspiracy theories and hoaxes abound when it comes to mankind’s arrival at our moon (or the non-arrival.) One such hoaxes is the Apollo 20, a purported top secret mission that back to the moon back in 1976 to fetch some aliens living there. The secret would have been revealed by no other than William Rutledge, who claims to have been an astronaut in that mission and who nowadays lives and writes out of Rwanda.

Arguably this theory was never as popular as its always popular theory that NASA spent billions of dollars to fabricate the moon landings but somehow forgot to paint stars in the ceiling.

Now however a film promises to inspire the masses who believe NASA has been sending secret missions to Luna. The movie is Apollo 18.

The movie’s premise is that after the Apollo program was cancelled, NASA would have flown another mission and found, of course, aliens and that’s why we’ve never been back to the Moon. It’s one more movie in the style of Cloverfield, where the audience is expected to pretend to be watching to real top-secret leaked footage.

Domension Films’ big cahuna, Bob Weinstein, stated,

We didn’t shoot anything, we found it. Found, baby!

The movie is of course a work of fiction. That’s not hard to figure out, even if you could not look up the actors who were in the movie. For instance, the astronauts of Apollo 18 in the movie are supposed to be Nathan Walker, John Grey e Benjamin Anderson, but the astronaut corps roster was very well known and none of these gentlemen were part of it. Although none of the crews for the cancelled flights (Apollos 18, 19 and 20) were never officially named – what would be the point? –, one can infer from the assignment rotation system NASA used that the Apollo 18 crew would likely be:

• Richard Gordon

• Vance Brand

• Harrison Schmitt

Except Schmitt was activated to the Apollo 17 main crew when it became clear that it would be the last chance for a scientist to step on the moon. Somebody – Joe Engle, perhaps – would have to replace him on Apollo 18.

But the hoax is not really about the movie itself, but about the general idea that NASA ran more missions than we know about. So, could something like Apollo 18 really have existed? Could NASA have performed this secretly?

It is unfortunately impossible to prove a negative, but at least we can think of how likely would that be. I can’t really see how such a thing could have been done. To begin with, there’s this:

You see, the Saturn V was big. Really big. Not easy to hide, then. It seems highly unlikely that NASA could have launched a Saturn V out of Cape Canaveral without it being seen.

I also saw this argument on some forum that NASA would have prefferred a night launch to improve the chances to keep it a secret, but the thing is, that big dumb rocket is not very subtle either.

And then if we discard a launch in the continental US, it would have to be from either a platform at sea or from somewhere in North Africa. Problem here is that the logistics of accomplishing such an feat – let alone in absolute secrecy – are just fenomenal.

As well, the Saturn V was a very public project. All its parts were very well tracked and it’s possible to know where most parts are even today. And some of those parts are huge, not the kind that you can stow in the back of a black unmarked van.

And then there are other factors, of course. We’re used to the image of three astronauts sitting on top of a rocket and a mission control room with, say, 20 or 30 people.

But an Apollo flight involved a lot more people than that. In A Man on the Moon: The Voyages of the Apollo Astronauts (Awesome gift! Thank you Stulzer!), Andrew Chalkin estimates the figure at 500,000 workers altogether. Others state the number is more like 400,000 people.

Regardless of the actual figure, it should be clear that such a mission would require the collaboration of hundreds of thousands of people all around the world (more on that below.)

And contrary to what some might want to believe, the Apollo program was not something entirely done behind closely guarded doors at some Air Force base. It involved a lot of private contractors. GE, IBM, Boeing, GM… the list of contractors can occupy several pages. To assume that all the employees involved who have since likely changed jobs multiple times and retired would be able to keep this a secret for four decades really stretches one’s imagination.

As big as the contractors’, the list of academic institutions involved in the program is amazing. Virtually every major US university and institute was included, but the list also included institutions form Australia, Belgium, Canada, England, Finland, Germany, Japan, Scotland and Switzerland. And that takes me to what is, to me, the most important thing to consider.

In order to fly to the Moon, you don’t just point the rocket, turn on the ignition and wait for it to reach its destination. It’s a complex voyage with huge preparations, calculations and adjustments. Orbital mechanics was one of the most interesting things I’ve even studied. But I digress.

An interesting challenge for NASA was to be able to communicate and track the ship all the way to the Moon and back.

Nowadays NASA added a whole network of satellites to assist the tast, but back when Mercury and Apollo were underway, the tracking depended on a network of tracking stations, vessels and aircraft all around the world.

That network was the Manned Space Flight (Tracking) Network. Starting from Apollo 10, the Deep Space Network was added to assist.

It’s interesting to know that over the course of its evolution, NASA’s networks included a station in Havana, Cuba. That station was dismantled after years of service due to the Cuban revolution. As Brazilian, I also find special interest in that there was even one station along with the Brasilia International Airport, even though it was quickly disassembled and shipped to Madagascar.)

Although those were NASA installations, they employed locals. These stations all around the world had been performing duties for years under heavy media scrutiny and to expect that all of a sudden they would be able to do it in complete secrecy is beyond reasonable belief. That and the local workers who also had to keep secrets for four decades make the most absurd hole in the hoax.

Now remember that the Apollo missions were tracked and monitored by several governments including, of course, the Soviet Union. In Two Sides of the Moon, Russian cosmonaut Alexei Leonov – trained to be the first man on the moon – states that all the missions were followed by the Russians in a very well equipped Space Transmissions Corps, in Moscow. Now you need the secret to be kept by the Soviets and then, after the USSR fell, by the several nations that sprouted from it.

And the Apollo missions were tracked by amateur astronomers and radio operators all over.

Of course that nothing here proves beyond any doubt that such secret missions were impossible, just unreasonable unlikely. But since you can’t prove a negative, no matter how improbale, there will still be plenty of people who believe.

I blamed the garbage collector for a slow parser this week. It was a satisfying theory because it required no evidence and made the runtime somebody else’s problem. Then I collected a CPU profile.

The programme only needed a profile file and a bounded run:

f, err := os.Create("cpu.prof")
if err != nil {
	return err
}
defer f.Close()

pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()

parseAll(files)

The exact profiling API and command-line tools are still moving with the snapshots, but the important method is stable enough: measure a representative workload, inspect where samples accumulate, change one thing, and measure again.

My surprise was a tiny character-class helper near the top of the profile. It performed a linear search through a string for every input byte. The allocator appeared, but it was not remotely the villain I had prepared a speech about. Replacing the repeated search with a direct table lookup changed the run far more than any speculative memory trick.

A CPU profile is sampling, not a complete diary. At intervals, the runtime records where execution is spending time. Enough samples reveal hot stacks without instrumenting every function call. That means very short runs can mislead, and blocked time is not the same as CPU time. A programme waiting on disk may feel slow while producing an uninteresting CPU profile.

Heap information answers another question: what remains live or what has been allocated, depending on the profile and tool. I have learned not to blur those together. A high allocation rate can increase collection work even if the live heap stays modest. A large live heap can be perfectly reasonable if the programme genuinely needs the data.

I prefer profiles to stopwatch calls scattered through the source. The profile keeps context: not just how long a region took, but which call paths led there. A small benchmark is still useful after the profile identifies the narrow piece worth isolating.

The caveat is representativeness. Profiling a toy input optimizes the toy. Debug output, cold caches, and a different machine can also distort results. I record the snapshot version and input beside the profile because both are part of the result.

The parser is faster now. More importantly, the garbage collector has been acquitted for lack of evidence, which is the closest a runtime component gets to an apology.

I reached for reflection after writing the same diagnostic print twice. The obvious first experiment was to inspect one value:

func describe(x interface{}) {
	v := reflect.ValueOf(x)
	fmt.Println(v.Type(), v.Kind())
}

What surprised me was that type and kind answer different questions. A named integer type has its own type, but its kind is still integer. The type preserves identity; the kind tells reflective code which family of operations is available.

An interface value passed to describe carries a dynamic type and data. reflect.ValueOf exposes a view of that pair. From there, operations are checked at run time. Asking for an integer from a string value is not a clever conversion; it is a panic with good timing.

Settable values were the second lesson. Reflecting on a copied value lets me inspect it, not rewrite the caller’s variable. To change the original I must pass a pointer, obtain the pointed-to value, and ensure it is settable. Reflection follows the same addressability rules as ordinary Go, only with more opportunities to discover mistakes while running.

I prefer ordinary interfaces whenever the required behaviour can be named. They give compile-time checking and explain intent better than a tour through Kind. Reflection earns its place when the types themselves are the input, as in formatting or decoding.

The caveat is that a generic-looking helper can become a private, badly documented type system. My diagnostic printer stayed reflective. The business logic did not. One magician in the programme is plenty.

Much has been said about the pros and cons of anonymity lately, prompted by Google+ TOS which require the use of one’s real name. No pseudonyms allowed, except apparently if you call yourself Lady Gaga or 50 Cent.

I have seen many kinds of arguments both for and against the use of aliases and I will not repeat them here. There is however one use of aliases that I haven’t seen stated anywhere and that coincidentally affects me personally. Perhaps this is so because the problem I am about to present is not so common after all. Or perhaps it is common but people decide not to talk about it. I have no way of knowing.

Anonymity is a vital necessity to people with a certain kind of disability, a mental disorder. I am such a person. As some of my friends know and others mock, I suffer from a mental condition called social phobia, also known as social anxiety. I take medications that help me overcome some of the most serious effects and that allow me to do things like write about it on this very blog.

Social anxiety manifests itself in varying degrees in all kinds of social interactions. And the levels of manifestations are not what you might expect. I regularly make presentations without a second thought. I’ve given talks to hundreds of people. And yet, ordering a pizza over the phone is terrifying experience to me. No matter how many times I’ve done it, I still have to “prepare” myself every time. I rehearse, play several unlikely scenarios in my head until I finally get the courage to dial the number and talk to the person on the other side. One characteristic of this anxiety disorder is that rationally I know that there is nothing wrong; there is no risk in calling the pizza place. But the brain acts as if there were. But I digress.

I love coding. I have been doing it since I was a kid and it’s the best thing I know how to do. And then there is open source. Open source projects should be the perfect venue for me to have fun. Except I am scared stiff by the idea that someone might laugh at the code. It came to a point where it is impossible for me to contribute. Then I’ve come up with a solution: an alias. For the past several years I’ve lived two different lives online: one as myself and another as an alias. I keep them strictly separate.

Using the alias, I actively contribute to several different projects. And I enjoy it all. And it would be impossible for me to do that using my own name. My pseudonym allows me to work around my condition. It allows me to live my life.

I understand the rationale behind the requirement for real names on Google+. But I also know that the requirement makes it impossible for people like me to be really free on the Internet. So far, Google hasn’t figured out my alias. Hopefully it never will.

(Photo by Abhishek Singh)