Capture colored console output in a tkinter window

The problem

I want to run a console script that emits ansi color codes to generate a colorful output and capture that colorful output in a tkinter window. Ideally I want this to work for programs like cmake or emerge which semi-intelligently switch off color codes if they detect they are not running on a terminal. This is all tested under linux. Some if it might work on windows too, but I haven't investigated.

Approach

I will show some increasingly sophisticated approaches using python 3.2 and try to outline their limits or problems. In order not to overload the presentation, this section details some functions that are used by the different approaches below. First of all in a file ansicolortext.py I have the following MIT licensed code. It is a tkinter Text widget that tranforms ansi color code sequences into tkinter color tags. Next, in a file timeout.py I have the following code, created by Matt Harrison under MIT license and posted as an ActiveState recipe: Finally, here's a function to create the user interface (a tkinter window with a resizable AnsiColorText widget)

Approach 1

If you google for this type of stuff almost always someone will come up with something as follows: This simple approach works beautifully for the example given (ls --color) and is highly recommended if it works for you. For some programs, however, this doesn't work. Many programs try to detect if they are running on a terminal or if they are part of a piped series of commands, and in the latter case suppress color codes. Examples of such programs include CMake and grep with the --color option. The end result is that your output shows up in black and white.

Approach 2, attempt 1

In order to fool programs like CMake or grep into producing color codes we need to pretend we are a terminal. The way to do this in linux is by using a pseudoterminal. According to wikipedia,

In some operating systems, including Unix, a pseudo terminal is a pseudo-device pair that provides a text terminal interface without an associated device, such as a virtual console, computer terminal or serial port. Instead, a process assumes the role of the underlying hardware for the pseudo terminal session.

Pseudoterminals can be accessed using Python's pty module. pty.openpty() opens a new pseudo-terminal pair and returns a pair of file descriptors (master, slave), for the master and the slave end, respectively. The terminal emulator process is associated with the master device and the shell is associated with the slave. The terminal emulator process (master) receives input from the keyboard and mouse using windowing events, and is thus able to transmit these characters to the shell (slave), giving the shell the appearance of the terminal emulator being an underlying hardware object. Any terminal operations performed by the shell in a terminal emulator session are received and handled by the terminal emulator process itself (such as terminal resizing or terminal resets). This approach looks clean and simple, but it has a problem: os.read blocks when no more data comes in, and so our script hangs before the UI can even come up. One proposal could be to use a timeout when reading data using os.read.

Approach 2, attempt 2

This approach now deals with the blocking: upon a timeout the communication is stopped. It suffers from a different problem though: the UI doesn't appear before all data has come in. In the case of a CMake script this can take minutes to hours to complete. Not exactly what we wanted... therefore we need a slightly smarter approach.

Approach 2, attempt 3

The idea is this: we will run all the communication in a thread of its own. The UI will run in the main thread. The communication thread will send the data it receives to the main thread by means of a Queue object. The main thread will read data from the Queue and display it in the text widget as it receives it.

The TextUpdater object T lives in the program's main thread. It reads data from the communication thread t by means of Queue q and sends it to the Text widget text, which itself is a child widget of the Tk root window root. The TextUpdater thread is asked to start working 100ms after root.mainloop() is started. Like that the UI has ample time to get itself ready for processing. The communication thread ThreadCommand t opens a pseudoterminal and uses it to communicate with the program being run. It stops itself and flags itself as not running anymore if no more data comes from the process (i.e. if the reading timed out). Reading data from the process uses a version of os.read with a timeout added to it (read2). The data read by read2 is slightly processed before it is sent to the communication thread: only complete lines of text are sent to avoid that we break up text in the middle of a color code (this would give trouble because the AnsiColorText widget isn't smart enough to deal with half-finished color codes).

If the timeout value used in read2 is too small, the timeout can occur too soon (especially with processes that take some time to generate output, like CMake) so this part is still a weak spot in this approach. Too long a timeout causes one to wait long after the process finishes to really close the communication thread. If you know of better ways to handle this, please comment! After text is written to the Text widget, we call root.update() to give Tk() the chance to handle pending events. This makes sure that the text widget is updated as input comes in, and it also makes sure that you can resize the window or scroll through the displayed data while data is still coming in.

An alternative without timeout decorator

As I found out today, there's an alternative without timeout decorater, by using the poll call of Python's select module. It still uses a timeout mechanism to decide if there's data to be read. This time the timeout mechanism is provided by the select module directly.

Screenshot

Here's a screenshot of CMake producing colored output in a Tk window. While CMake is running, the window can be resized and you can navigate through it with arrow/pageUp/pageDown/... keys.