I’ve been confused about what’s going on with terminals for a long time.

But this past week I was using xterm.js to display an interactive terminal in a browser and I finally thought to ask a pretty basic question: when you press a key on your keyboard in a terminal (like Delete, or Escape, or a), which bytes get sent?

As usual we’ll answer that question by doing some experiments and seeing what happens :)

remote terminals are very old technology

First, I want to say that displaying a terminal in the browser with xterm.js might seem like a New Thing, but it’s really not. In the 70s, computers were expensive. So many employees at an institution would share a single computer, and each person could have their own “terminal” to that computer.

For example, here’s a photo of a VT100 terminal from the 70s or 80s. This looks like it could be a computer (it’s kind of big!), but it’s not – it just displays whatever information the actual computer sends it.

Of course, in the 70s they didn’t use websockets for this, but the information being sent back and forth is more or less the same as it was then.

(the terminal in that photo is from the Living Computer Museum in Seattle which I got to visit once and write FizzBuzz in ed on a very old Unix system, so it’s possible that I’ve actually used that machine or one of its siblings! I really hope the Living Computer Museum opens again, it’s very cool to get to play with old computers.)

what information gets sent?

It’s obvious that if you want to connect to a remote computer (with ssh or using xterm.js and a websocket, or anything else), then some information needs to be sent between the client and the server.

Specifically:

• the client needs to send the keystrokes that the user typed in (like ls -l)
• the server needs to tell the client what to display on the screen

Let’s look at a real program that’s running a remote terminal in a browser and see what information gets sent back and forth!

we’ll use goterm to experiment

I found this tiny program on GitHub called goterm that runs a Go server that lets you interact with a terminal in the browser using xterm.js. This program is very insecure but it’s simple and great for learning.

I forked it to make it work with the latest xterm.js, since it was last updated 6 years ago. Then I added some logging statements to print out every time bytes are sent/received over the websocket.

Let’s look at sent and received during a few different terminal interactions!

example: ls

First, let’s run ls. Here’s what I see on the xterm.js terminal:

bork@kiwi:/play$ ls
file
bork@kiwi:/play$

and here’s what gets sent and received: (in my code, I log sent: [bytes] every time the client sends bytes and recv: [bytes] every time it receives bytes from the server)

sent: "l"
recv: "l"
sent: "s"
recv: "s"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
recv: "file\r\n"
recv: "\x1b[?2004hbork@kiwi:/play$ "

I noticed 3 things in this output:

1. Echoing: The client sends l and then immediately receives an l sent back. I guess the idea here is that the client is really dumb – it doesn’t know that when I type an l, I want an l to be echoed back to the screen. It has to be told explicitly by the server process to display it.
2. The newline: when I press enter, it sends a \r (carriage return) symbol and not a \n (newline)
3. Escape sequences: \x1b is the ASCII escape character, so \x1b[?2004h is telling the terminal to display something or other. I think this is a colour sequence but I’m not sure. We’ll talk a little more about escape sequences later.

Okay, now let’s do something slightly more complicated.

example: Ctrl+C

Next, let’s see what happens when we interrupt a process with Ctrl+C. Here’s what I see in my terminal:

bork@kiwi:/play$ cat
^C
bork@kiwi:/play$

And here’s what the client sends and receives.

sent: "c"
recv: "c"
sent: "a"
recv: "a"
sent: "t"
recv: "t"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
sent: "\x03"
recv: "^C"
recv: "\r\n"
recv: "\x1b[?2004h"
recv: "bork@kiwi:/play$ "

When I press Ctrl+C, the client sends \x03. If I look up an ASCII table, \x03 is “End of Text”, which seems reasonable. I thought this was really cool because I’ve always been a bit confused about how Ctrl+C works – it’s good to know that it’s just sending an \x03 character.

I believe the reason cat gets interrupted when we press Ctrl+C is that the Linux kernel on the server side receives this \x03 character, recognizes that it means “interrupt”, and then sends a SIGINT to the process that owns the pseudoterminal’s process group. So it’s handled in the kernel and not in userspace.

example: Ctrl+D

Let’s try the exact same thing, except with Ctrl+D. Here’s what I see in my terminal:

bork@kiwi:/play$ cat
bork@kiwi:/play$

And here’s what gets sent and received:

sent: "c"
recv: "c"
sent: "a"
recv: "a"
sent: "t"
recv: "t"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
sent: "\x04"
recv: "\x1b[?2004h"
recv: "bork@kiwi:/play$ "

It’s very similar to Ctrl+C, except that \x04 gets sent instead of \x03. Cool! \x04 corresponds to ASCII “End of Transmission”.

what about Ctrl + another letter?

Next I got curious about – if I send Ctrl+e, what byte gets sent?

It turns out that it’s literally just the number of that letter in the alphabet, like this:

• Ctrl+a => 1
• Ctrl+b => 2
• Ctrl+c => 3
• Ctrl+d => 4
• …
• Ctrl+z => 26

Also, Ctrl+Shift+b does the exact same thing as Ctrl+b (it writes 0x2).

What about other keys on the keyboard? Here’s what they map to:

• Tab -> 0x9 (same as Ctrl+I, since I is the 9th letter)
• Escape -> \x1b
  
• Backspace -> \x7f
• Home -> \x1b[H
• End: \x1b[F
• Print Screen: \x1b\x5b\x31\x3b\x35\x41
• Insert: \x1b\x5b\x32\x7e
• Delete -> \x1b\x5b\x33\x7e
• My Meta key does nothing at all

What about Alt? From my experimenting (and some Googling), it seems like Alt is literally the same as “Escape”, except that pressing Alt by itself doesn’t send any characters to the terminal and pressing Escape by itself does. So:

• alt + d => \x1bd (and the same for every other letter)
• alt + shift + d => \x1bD (and the same for every other letter)
• etcetera

Let’s look at one more example!

example: nano

Here’s what gets sent and received when I run the text editor nano:

recv: "\r\x1b[Kbork@kiwi:/play$ "
sent: "n" [[]byte{0x6e}]
recv: "n"
sent: "a" [[]byte{0x61}]
recv: "a"
sent: "n" [[]byte{0x6e}]
recv: "n"
sent: "o" [[]byte{0x6f}]
recv: "o"
sent: "\r" [[]byte{0xd}]
recv: "\r\n\x1b[?2004l\r"
recv: "\x1b[?2004h"
recv: "\x1b[?1049h\x1b[22;0;0t\x1b[1;16r\x1b(B\x1b[m\x1b[4l\x1b[?7h\x1b[39;49m\x1b[?1h\x1b=\x1b[?1h\x1b=\x1b[?25l"
recv: "\x1b[39;49m\x1b(B\x1b[m\x1b[H\x1b[2J"
recv: "\x1b(B\x1b[0;7m  GNU nano 6.2 \x1b[44bNew Buffer \x1b[53b \x1b[1;123H\x1b(B\x1b[m\x1b[14;38H\x1b(B\x1b[0;7m[ Welcome to nano.  For basic help, type Ctrl+G. ]\x1b(B\x1b[m\r\x1b[15d\x1b(B\x1b[0;7m^G\x1b(B\x1b[m Help\x1b[15;16H\x1b(B\x1b[0;7m^O\x1b(B\x1b[m Write Out   \x1b(B\x1b[0;7m^W\x1b(B\x1b[m Where Is    \x1b(B\x1b[0;7m^K\x1b(B\x1b[m Cut\x1b[15;61H"

You can see some text from the UI in there like “GNU nano 6.2”, and these \x1b[27m things are escape sequences. Let’s talk about escape sequences a bit!

ANSI escape sequences

These \x1b[ things above that nano is sending the client are called “escape sequences” or “escape codes”. This is because they all start with \x1b, the “escape” character. . They change the cursor’s position, make text bold or underlined, change colours, etc. Wikipedia has some history if you’re interested.

As a simple example: if you run

echo -e '\e[0;31mhi\e[0m there'

in your terminal, it’ll print out “hi there” where “hi” is in red and “there” is in black. This page has some nice examples of escape codes for colors and formatting.

I think there are a few different standards for escape codes, but my understanding is that the most common set of escape codes that people use on Unix come from the VT100 (that old terminal in the picture at the top of the blog post), and hasn’t really changed much in the last 40 years.

Escape codes are why your terminal can get messed up if you cat a bunch of binary to your screen – usually you’ll end up accidentally printing a bunch of random escape codes which will mess up your terminal – there’s bound to be a 0x1b byte in there somewhere if you cat enough binary to your terminal.

can you type in escape sequences manually?

A few sections back, we talked about how the Home key maps to \x1b[H. Those 3 bytes are Escape + [ + H (because Escape is \x1b).

And if I manually type Escape, then [, then H in the xterm.js terminal, I end up at the beginning of the line, exactly the same as if I’d pressed Home.

I noticed that this didn’t work in fish on my computer though – if I typed Escape and then [, it just printed out [ instead of letting me continue the escape sequence. I asked my friend Jesse who has written a bunch of Rust terminal code about this and Jesse told me that a lot of programs implement a timeout for escape codes – if you don’t press another key after some minimum amount of time, it’ll decide that it’s actually not an escape code anymore.

Apparently this is configurable in fish with fish_escape_delay_ms, so I ran set fish_escape_delay_ms 1000 and then I was able to type in escape codes by hand. Cool!

terminal encoding is kind of weird

I want to pause here for a minute here and say that the way the keys you get pressed get mapped to bytes is pretty weird. Like, if we were designing the way keys are encoded from scratch today, we would probably not set it up so that:

• Ctrl + a does the exact same thing as Ctrl + Shift + a
• Alt is the same as Escape
• control sequences (like colours / moving the cursor around) use the same byte as the Escape key, so that you need to rely on timing to determine if it was a control sequence of the user just meant to press Escape

But all of this was designed in the 70s or 80s or something and then needed to stay the same forever for backwards compatibility, so that’s what we get :)

changing window size

Not everything you can do in a terminal happens via sending bytes back and forth. For example, when the terminal gets resized, we have to tell Linux that the window size has changed in a different way.

Here’s what the Go code in goterm to do that looks like:

syscall.Syscall(
    syscall.SYS_IOCTL,
    tty.Fd(),
    syscall.TIOCSWINSZ,
    uintptr(unsafe.Pointer(&resizeMessage)),
)

This is using the ioctl system call. My understanding of ioctl is that it’s a system call for a bunch of random stuff that isn’t covered by other system calls, generally related to IO I guess.

syscall.TIOCSWINSZ is an integer constant which which tells ioctl which particular thing we want it to to in this case (change the window size of a terminal).

this is also how xterm works

In this post we’ve been talking about remote terminals, where the client and the server are on different computers. But actually if you use a terminal emulator like xterm, all of this works the exact same way, it’s just harder to notice because the bytes aren’t being sent over a network connection.

that’s all for now!

There’s defimitely a lot more to know about terminals (we could talk more about colours, or raw vs cooked mode, or unicode support, or the Linux pseudoterminal interface) but I’ll stop here because it’s 10pm, this is getting kind of long, and I think my brain cannot handle more new information about terminals today.

Thanks to Jesse Luehrs for answering a billion of my questions about terminals, all the mistakes are mine :)

Security-Enhanced Linux (SELinux) is a type of Mandatory Access Control (MAC) in the Linux kernel. It can prevent software from performing unexpected — such as abusive or malicious actions — on your Linux systems. However, … it’s also an unmanageable mess, and I have a much greater understanding of why people recommend that people disable it.

Read more …

by Daniel A. Kaufman

___

It is a strange thing to stare down at a box in a hole in the ground and think that your father is inside it, but that’s exactly what I did last Monday, when we buried my dad, Alexander Kaufman, who would have been 94 years old this June.

I called him “Aba,” which is “father” in Hebrew, and funnily enough, when I was in graduate school, my friends called him that too when speaking of and about him. My daughter called him “Pop Pop.” To everyone else, he was “Alex.”  

Aba was an enthusiastic and expert photographer who took pictures of everything and saved every picture he acquired. When I was younger, I found this annoying, and I recall my mother disliking it as well. Thinking about it today, though, he clearly was right. To have such a substantial, high-quality photographic record not just of our lives – my father’s, mother’s and my own, that is – but of the lives of our extended family, going back as far as the 19^th century, is a remarkable and rare thing. And in light of these last few years, where what I saw of my father just seemed to get worse every time, these photos offer a strong, persistent reminder that he lived a long, accomplished, and satisfying life; that his time here was good. 

And what a life it was! Aba and his family were driven out of Nazi Germany when he was a young child. He grew up in Mandatory Palestine where, as a young adolescent, he joined the Haganah and later fought in the 1948 Independence War that created the State of Israel. (At one point, he even found himself charged with delivering the entirety of the fledgling nation’s currency to a bank in Tel Aviv, which he recounts in a humorous story that appears in his book, Time Flies.) Palestine/Israel was also where he courted my mother, though they would be married in Switzerland, as Jews could only hold Orthodox weddings in Israel back then, and neither of my parents wanted that.  

My father moved to New York just a few years after the war (my mother followed not long after), where he eventually founded and managed Displaycraft, a design company that would be the source of my family’s income over the course of my childhood and adolescence. Displaycraft produced displays and exhibits for the travel industry, museums, galleries, and even, strangely enough, prefabricated housing for the “insta-towns” needed by the oil industry, in places like Saudi Arabia and pre-revolutionary Iran. Entrepreneurial down to his core, my father had figured out that the extrusion system he used to create displays and exhibits could also be used for housing that could be quickly and easily built and just as easily dismantled.    

But his greatest achievement in this capacity was a permanent exhibition commissioned to open the then-new Diaspora Museum in Tel Aviv, in 1978. Originally called Beit Hatfutsot, it is now the Museum of the Jewish People, and for its opening, my father’s shop produced 21 scale models (interiors and exteriors) of historically significant synagogues from around the world that are meticulous in their detail, down to every chandelier, chalice, and chair. The research, which involved traveling the world, and photographing scores of synagogues, inside and out, took well over a year, and is chronicled in his book Trawling Twenty Centuries. 

I was born in 1968, and my parents raised me in a lovely house in Roslyn, NY, a village on Long Island’s North Shore that predates the American Revolution. Ours was a traditional, upper middle-class American household of the early to mid-1970’s, with my father commuting daily to New York City to work, and my mother being a homemaker. This described the family situation of most of my friends as well, as the divorce and latchkey kid wave that was to come was a feature of late- rather than early-Gen X life. 

Aside from family vacations, then, which were plentiful, I only saw my father on nights and weekends, and sometimes not even then if he had to work late or was travelling on business, as he often did. Nonetheless, I felt his love and presence strongly, in part because he transformed the entire basement of our house into a sophisticated playroom/art studio, where I spent a good portion of my childhood, and where I was reminded constantly of how much Aba cared about me. And as I got older and became involved in competitive baseball and tennis, my father was my biggest enthusiast and fan.

By the time I graduated high school and went to the University of Michigan in 1986, Aba had bought the building in which his business was located and sold Displaycraft and its related ventures. Increasingly hostile relations with the labor unions in New York City and the realization that he was making more money as a commercial landlord than as a designer and builder of displays and exhibits were what convinced him to get out. From the 1990’s on, his primary business endeavors would be in the areas of commercial real estate and finance and doing this allowed him to amass a considerable amount of wealth.  

This also was the period when he embarked upon several independent creative endeavors. He began to write books, the most substantial and impressive of which is the massive Precipice Option, which tells the story of Rezső Kasztner and how he saved my mother and her family as well as thousands of Hungarian Jews, during the Holocaust. Aba also began a long running cartoon series, Sal and Al, though I always thought the art was better than the writing. Most significantly, he developed a novel system of painting, in which he would create an image or design in paint, scan it, and then reproduce it in strips and patches of colored vinyl that would be applied to canvas, producing a very modern, sleek, almost Pop-Art body of work. Aba made scores of these under the name “Xela.” 

My father was not made for old age. Few people are, of course, but in his case, the fit between his personality and the physical debilitation that would define his last few years was about the worst one could imagine.  

Aba’s upbringing and formative experiences had created a man who always was outwardly and materially directed and had little by way of what I might call an “inner life.” It wasn’t that he didn’t feel and experience things strongly – he most certainly did – but rather that he wasn’t contemplative or thoughtful when alone with himself. Even his creative energies required not just material realization but commercial application. I can derive great satisfaction from merely entertaining interesting ideas and projects or drafting sketches and outlines of things, or just reading and contemplating, without ever following up on any of it. My father, however, was a man of action. He was driven to create; to make; and to sell.   

Moving out of his artistically creative business ventures into commercial real estate and finance was the worst move he could have made … for him, that is. Yes, it guaranteed financial security for our family, for which I will be eternally grateful, but the work was too easy for him and demanded little of his creative and energetic mind. The paintings and books were supposed to satisfy this part of him, but it was here that the entrepreneurial sensibility that so characterized Aba turned out to be a liability. To really make it either in commercial fine arts or book publishing, one has to work within institutional constraints and with handlers and minders. My father would accept no editors, however, and refused to play the socialite games required of aspirant professional artists. The result was that the books have an amateurish quality to them (with the exception of the last, Startups, which, due do to the advanced state of his illness, he allowed me to edit), and the paintings never really managed to penetrate the market, though he did have a few gallery exhibitions and some impressive individual sales. My father was doing these things in order to satisfy his creative and imaginative needs and to challenge himself, but the way he went about them guaranteed that they never would never achieve the commercial success he needed in order for an endeavor to be gratifying. The only thing that really gave him joy in the last few years was my daughter, Victoria. 

His final years also helped me to understand the importance of a certain kind of formal education. Given the circumstances of his early life, my father dropped out of school very young, which meant that he had no real formal education in specific disciplines and subject areas. People today who are not scientists or in scientifically oriented professions like Engineering often think that they got nothing out of their science educations, but this is untrue in an important sense. The science education I received as an adolescent was sufficient to ensure that I understand basic things about illness, disease, and the human body that make navigating modern health care possible. It’s why I understand what my doctors tell me and why I am inclined to do what they say. My father lacked this education and understanding, and it very much showed in the disastrous way he handled the last stages of his illness. He refused to move into a managed environment and fought every effort I made to marshal more support in the house. Only after his fourth or fifth hospitalization in a span of three years – all of which almost led to death – did he finally accept home hospice, but by this point, it was too late for him to salvage any satisfying or rewarding moments from what little time remained. Certainly, some of this was due to an irascible personality, but in good part, it was a function of ignorance of the mechanics and imperatives of the human body and an overestimation of the efficacy of the will on that body. 

Aba’s final months were bad, and there is no point in lying about or sugar-coating it. But they were a mere moment in a remarkable life that spanned almost the entire twentieth century and a quarter of the twenty first. My father was very much a man of that era and of a type that only really could exist and flourish in that era; a type that has largely disappeared and to our detriment.

He also was a wonderful father to me, and an even more wonderful grandfather to my daughter. And that is how I choose to remember him. I love you Aba.