Super Commuting One Year Later

I've been super commuting from Los Angeles to Oracle HQ in Redwood City for a little over a year, and it's been a fascinating ride. I've learned so much more about flying, aviation, delays, work/life balance, my body, and remote work than I've ever imagined. I fly from LAX to the bay area every Tuesday morning and fly back down every Wednesday night and spend the other three days working from home. What have I learned?

First, I'm astonished at how flexible and adaptable my body is. The first time I flew, I was dreading the commute. I had to wake up at 5 AM to be at work by 10 AM. I drearily stood in TSA lines, unpacking my laptop, taking out my liquids, and sitting in between two also dreary travelers. It didn't quite have the sexiness some people imagine with flying, nor did it invigorate me. It was a energy sucking experience.

A year later, I actually enjoy flying. There are of course a few perks with having elite status on an airline. I no longer sit in TSA lines thanks to TSA Pre, which requires a traveler to be an airline frequent flyer. But this means I can zip through TSA lines in less than a minute because you don't even have to take off your belt or laptop! You just walk through security. As an elite, you get economy plus seats, which is plenty of space for a shorter person like myself. On the actual flight myself, I just read my kindle as if I was sitting on the subway. My brain has post processed all the airplane engine noise that I don't even notice it anymore. Half of the time, I can actually sleep on the plane too, so to me it feels like a normal one hour subway ride. I'm quite surprised at how quickly my body has adjusted to flying.

There were lots of battles, mostly battling San Francisco Airport (SFO). SFO is plagued by delays and is the most delayed airport in August due to fog. Essentially the problem is that SFO has two runways for take off and landing in an X figure. On good days, two planes land in parallel followed by two planes departing in parallel. The problem is that for parallel landings, the pilots of both airplanes must be able to see each other (makes sense). However, because SFO has fog, the FAA has regulated that runways must be 2,500 feet apart to do instrument based landings. SFO's runways are only 750 feet apart, essentially closing down one runway during fog. This can lead up to 3 hours of delays by the evening due to a cascade of delays. Protip: fly through SJC. I've also learned optimal flight schedules and flight prices due to weather patterns between LAX and SJC. LAX has fog and has minor delays in late May through mid July. LAX to the bay area in winter should cost ~$140 roundtrip and in summer ~$175. Booking between 3 - 6 weeks in advance give you the best prices.

The other half of flying is that working from home, totally remote all year is really difficult. There are times when it's fantastic to be remote, such as a morning bike ride to the beach. The other half is draining and it makes it difficult to make and maintain work relationships. I think of Oracle was a fully distributed company, it'd be fine, but every week is difficult. I think most people are better off easing into remote work, or making occasional trips to the office. I'm actually quite happy that LA and SF are in the same time zone and is a short flight away. It makes it rather inexpensive to go to visit HQ and get face time. I think optimally, every other week is probably enough, but I don't mind every week.

Overall, I'm quite happy and surprised at how well Super Commuting is working out. I think it's been one of the best decisions I've made last year and hope to continue doing it.

Adventures in Flooring

I first ventured into the realm of home improvement by mistake. I was cleaning my new fangled condo with 409 and Baking Soda, the enemy of dirty grout lines. As I scrubbed, I destroyed the sealant between the shower and the floor, revealing quite a few interesting things. First, the floor started to curl up, which made me confused. Wasn't floor supposed to be wood or tile,  how could it possibly curl up? I pulled back the floor some more and saw that the floor was actually like wall paper! Next, when I stepped onto the area where the floor was curling, I would sink into the floor, much like stepping into a bed. What was going on! Into the rabbit hole we go.

Apparently floors have three layers The top layer, what you see, can be made of multiple different materials. The first layer is what you step on when you go take a shower. It can be either Vinyl, Hardwood, or Tile. One layer underneath that, the middle layer, is what your top layer rests on. Usually this is plywood or backer board. Lastly, underneath all that is the subfloor, which hooks into joists, which actually hold up the building. What actually happened to my floor was that my 409 / Baking Soda simply unveiled all the damage underneath and into the middle layer. Here's a diagram of a floor:

The middle layer, made of plywood, which normally held up my floor was corroded by mold and water. My simple $50 fix became a $1000 fix, oh yay for home ownership! Here's a picture of the damaged floor:

There are a few fixes depending on how much money you want to spend, and like the n00b I was, I spent a lot of time in Home Depot dazed and confused. BTW, Home Depot's customer service is amazing, they are really patient and used to stupid people looking at a piece of equipment with a lost gaze. This is what normal people feel like when they walk into Frys!

The fixes are two fold, assuming the sub floor (the bottom layer), isn't damaged. Because bathrooms have lots of water, installing plywood is the "cheap" choice, but the plywood will eventually become water soaked and damaged over time. Have you ever hiked in the mountains and seen water soaked branches that are fragile and destroyed? That happens to your floor! You can buy redguard, which coats the plywood with a rubbery overcoat that resists the water, leaving a pool on top of the plywood preventing damage from the water. However, the water has to go somewhere and it just soaks into your top layer. If your top layer is water proof, this might not be a problem, but then you need tile (which we'll get into).

Another option, is laying down cement, which servers the same function as plywood but with cement instead. This protects your subfloor but you still have the water on the top layer problem. What Home Depot recommended was Backer Board, which absorbs water preventing it from falling down. Now you can buy these pieces of backer board and put water resistant tape between the seams of the backer board, creating a water absorbing middle layer. Now we have a middle layer, what to put on top?

There seems to be four options depending on how much money you want to spend. First, you can install natural hardwood, but that absorbs water and is a bad choice for kitchens and bathrooms. Second, you can install vinyl or laminoid, which is essentially wallpaper for your middle layer. The materials can be water resistant, preventing water from soaking into the middle layer at the high end of vinyl. The third option is tile, which is water proof, but is more expensive to install. Each tile piece has to be cut, laid down, and filled with grout (which is a pain to clean!) between each piece. And finally, you can install natural stone if you just IPOed your company.

I went with option two, installing vinyl on the top layer. What we had and originally caused the problem was a layer of vinyl on top of the middle layer of plywood, followed by the subfloor. The plywood was damaged by moisture, which happens in every bathroom. We didn't touch the subfloor. We replaced the middle layer with water absorbing backer board and put a layer of vinyl on top. Here's the subfloor after throwing the other layers away:

Step one, how do we install the backer board onto the sub floor? First, you have to saw the backer board into the correct measurements for your bathroom. My bathroom involved two pieces of backer board, sealed between them with some water sealant. Each backer board is screwed into the subfloor with some nails. Next, you use the sealant along all the edges of the bathroom. The sealant prevents water from going into the subfloor along the small seams at the edge between the wall and the backer board.

Step two, how do we install the vinyl on top of the backer board? I bought this floor that claims to be the easiest floor ever to install. Each vinyl piece is like a puzzle that snaps together to form a whole floor. Each piece is cut using a utility knife. What really happens is that each vinyl piece is water resistant and made of very strong material, so strong that it had to be cut with a saw. The actual layering of each piece to snap together worked fairly well, but the whole floor is "floating". There is adhesive behind each vinyl piece that sticks the vinyl to the backer board, but there is nothing solid holding the floor together like there is with tile. I've read reports that after a while, the floor's joints become weak and there are open seams between the vinyl pieces. We'll see what happens soon.

Step three, can I finally shower? Once the vinyl is put on top of the backer board, you seal up the edges of the bathroom to prevent water from dripping below the vinyl. Thus, each layer of the floor except the subfloor are individually sealed along the edges for water protection. The sealant takes a while to dry, so I was told to not use the bathroom for a few days. I waited 3 days and now I can finally shower!

El Fin

Becoming a Technical Leader

I forget where I stumbled upon this book, but I'm happy I did. Most leadership books have a tone of superiority, almost with a sense of being talked down to. What I liked most about this book was that it was easy to read and understand, while at the same time imparting valuable lessons and bringing up interesting questions.

Instead of the usual leadership book theme of lead the charge, understanding organizational issues and how to hire people, Becoming a Technical Leader actually starts with the reader. It asks the question, why do you, as a person, want to become a leader? What makes you actually valuable to an organization and why would anyone in their right mind actually follow or listen to anything you say? Every chapter has some central tenet that actually forces the reader to become introspective, with a list of questions at the end of the chapter to reinforce and clear out ideas. It's like Math homework without a correct answer in the back to the chapter.

For example, one chapter examines how people grow in skill and as a person. We all like to imagine that growth if linear, like y = mx + b, or if for some period of time, exponential growth. I felt the exponential growth curve when I first entered graduate school, feeling like a complete idiot  but at the same time learning so much everyday that I was on overdrive. The first time I was ever able to build a JIT compiler, assembling the add eax, ebx instruction was such a miracle. It felt exciting to learn so much and accomplish so much. However, by the time I wrote my fourth JIT compiler, I felt a plateau, and needed some stimulus to find new challenges. This book details the growth trajectory so I learned what signals to look out for at each step of the growth stage. 

However, the best part of the book is simply how easy it is to learn something about myself. When some discussions come up, I become defensive and have to become aware of my defense and shut it down. Others, like a good friend, disagreements come easy and no hard feelings erupt. It's simply a trained muscle of socially grace. That's what's so amazing about this book. I respect the author as a leader through his writing, as he tries to give me constructive feedback on how to improve myself. 

That's really the crux of the book. Instead of being a another outward looking leadership book, this book forces introspection on why and what to improve in yourself to become a leader. Not only did I learn about leadership skills, I learned a lot about myself. Highly recommended.

Solaris 11 Colored Terminal

I'm starting to do some development with Solaris, and while most of it is linux like, some things are still a little different. I spent a good 20 minutes trying to figure out how to add colors to the terminal output. No colors = No fun! Most of the googling showed a large array of export LS_COLORS="color combination", but that didn't work. All my previous .bashrc files that worked on Linux didn't work on Solaris. In addition, ls --color=auto didn't even show any colors. I was confused!

Apparently all I needed to add was:

export TERM=xterm-color

Now I get all my colorful glory in Solaris!

Thread Local Storage in Object Files

I was rummaging through some generated x86-64 assembly compiled by Clang and Intel's C compiler (ICC), and I noticed something strange. In Clang generated code, there are a large number of loads from a _DYNAMIC section.

1	6a1: lea 0x2002d7(%rip),%rdi # 200980 <_DYNAMIC+0x1a8>

The assembly didn't bother me, so much as the comment. What is the _DYNAMIC section doing? I've never seen anything like it before and it made me wonder. This was especially confusing because these instructions didn't seem to occur with the code compiled by ICC. What is that Dynamic section?

At first, I thought it was simply a case of position independent code (PIC) because the load was offset from the instruction pointer (%rip). I don't remember explicitly enabling position independent code, but I thought maybe Clang compiles PIC code by default. But something else struck me as odd. The ICC generated code had loads relative from %rip as well, but no comment. Maybe I just forgot to build the binary with debug information and that would add the _DYNAMIC comment?

I recompiled the binary with debug information with ICC but that didn't do it either. I knew that there was probably some issue with Thread Local Storage because I've experienced it before. I set out to discover what it was. Let's have some simple C code with a thread local variable and see what happens.

1 2 3 4 5 6

__thread int _threadInt; int foo() { printf("Thread int is: %d\n", _threadInt); return 0; }

Clang generates the following assembly code:: 

1 2 3 4 5 6 7 8

push %rax mov %fs:0xfffffffffffffffc,%esi mov $0x400658,%edi xor %al,%al callq 4003f0 <printf@plt> xor %eax,%eax pop %rdx retq

Here, instruction #2 is the most important. This is a thread local variable access. The generated assembly uses the initial-exec storage model, which is another blog post. Instruction #5 is the call to printf. Here the assembly makes sense. We just load the thread local variable and pass it as a parameter into printf. (X86-64 passes parameters in registers). But there's still no load from _DYNAMIC! What's going on? I read more about the DYNAMIC section in ELF files [1], trying to figure out where the thread local variable is allocated. Let's take a look at the DYNAMIC section in the object file:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

objdump -x main.o Dynamic Section: NEEDED libc.so.6 NEEDED ld-linux-x86-64.so.2 INIT 0x0000000000000568 FINI 0x0000000000000718 GNU_HASH 0x00000000000001b8 STRTAB 0x0000000000000360 SYMTAB 0x00000000000001f8 STRSZ 0x00000000000000b6 SYMENT 0x0000000000000018 PLTGOT 0x0000000000200990 PLTRELSZ 0x0000000000000060 PLTREL 0x0000000000000007 JMPREL 0x0000000000000508 RELA 0x0000000000000478 RELASZ 0x0000000000000090 RELAENT 0x0000000000000018 VERNEED 0x0000000000000438 VERNEEDNUM 0x0000000000000002 VERSYM 0x0000000000000416 RELACOUNT 0x0000000000000001

Here the dynamic section refers to elements that are required during dynamic linking. However, looking at this, we see some sections in ELF describing what's needed at runtime. We need libc for printf. We also see a symbol table (Item 10), but still nothing here that tells us how the thread local variable is accessed. In addition, the generated code doesn't have a comment indicating a load from a DYNAMIC section. But this gave me an idea, what if we made the thread local variable extern so the linker would have to resolve the address later? And since it should be in the dynamic section, let's make the code a shared library. Let's write some code:

1 2 3 4 5 6 7 8

clang -fpic main.c -O -shared -o main.o extern __thread int _threadInt; int foo() { printf("Thread int is: %d\n", _threadInt); return 0; }

1 2 3 4 5 6 7 8 9 10 11

push %rax lea 0x20031f(%rip),%rdi # 2009a8 <_DYNAMIC+0x1a8> callq 598 <__tls_get_addr@plt> mov (%rax),%esi lea 0xac(%rip),%rdi # 746 <_fini+0xe> xor %al,%al callq 578 <printf@plt> xor %eax,%eax pop %rdx retq nopw %cs:0x0(%rax,%rax,1)

Alright, finally! We have a reference to the DYNAMIC section again! Why did this happen? Compilers can optimize Thread Local Storage using different thread local storage models. One optimization is if the compiler can determine that the thread local variable is only referenced within the executable, a level of indirection can be optimized away, hence the lack of a _DYNAMIC access. By making the binary a shared library, we prevented the compiler from performing this optimization. So now that we finally have a reference to _DYNAMIC, what is it actually referencing? We see something interesting about instruction #3, a call to tls_get_addr@plt. 

This is one method of accessing thread local storage [2]. The code calls a function that retrieves the address of the local variable through the Procedure Linking Table (PLT). How the PLT works is another long blog post. Essentially, thread local access uses a table to patch up relocation points to access variables. Code accesses the variable through this table. Then I thought, oh let's take a look at the dynamic relocation entries in the object file:

1 2 3 4 5 6 7 8 9 10

readelf -r main.o Relocation section '.rela.dyn' at offset 0x478 contains 6 entries: Offset Info Type Sym. Value Sym. Name + Addend 0000002007f8 000000000008 R_X86_64_RELATIVE 00000000002007f8 000000200990 000300000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0 000000200998 000400000006 R_X86_64_GLOB_DAT 0000000000000000 _Jv_RegisterClasses + 0 0000002009a0 000500000006 R_X86_64_GLOB_DAT 0000000000000000 __cxa_finalize + 0 0000002009a8 000700000010 R_X86_64_DTPMOD64 0000000000000000 _threadInt + 0 0000002009b0 000700000011 R_X86_64_DTPOFF64 0000000000000000 _threadInt + 0

Aha! There it is, Line #9. If we look at the comment in the disassembled code, we see that it states an entry in the _DYNAMIC section pointing to address 0x2009a8. When we look at the dynamic relocation section, we see our thread local variable at the exact offset. 

Whew, at least now I know what the _DYNAMIC section is for. If refers to the dynamic relocation section in an ELF file, not the dynamic shared library section. Another blog post will have to be written to describe how relocation works and each thread local variable access optimization strategies.

References:
[1] Executable and Linkable Format
[2] ELF Handling for Thread Local Storage
[3] Eli Bendersky's Position Independent Code