Thursday, November 29, 2012
Saturday, October 13, 2012
Musings on the disaster-formerly-known-as-worldwide-market-leader Nokia
"Stephen Elop" is an anagram of "Phone Pestle". Just sayin'...
Sunday, September 16, 2012
Variations on producer consumer in C++
Image made by Stefaan Himpe in libreoffice draw, available under CC-BY-SA license.
Problem
Paraphrased from wikipedia:The consumer producer problem is a classical example of a multi-process synchronization problem. The problem describes two processes, the producer and the consumer, who share a common buffer used as a queue. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time, the consumer is consuming the data (i.e., removing it from the buffer and doing something with it) one piece at a time.As an example consider a printer server. The printer server receives requests to print long documents. We don't want the printer server to refuse new requests while the printer is still printing. Therefore we run it in a separate thread. After receiving the request, the printer server queues it in a printer queue and immediately returns to listening for new requests. The printer server is a producer. An application monitors the printer queue and prints the requested documents, one by one. The application is a consumer.
Challenges
- I still have too little practical experience in multithreaded programming. This article is written as part of my investigation into what is possible. If you spot mistakes or inaccuracies, please comment!
- How do we push and pop work items on/from a queue in a multi-threaded environment?
- If the producer is faster than the consumer, how do we prevent the queue from exploding in size, eventually filling up all available memory?
- How can a consumer wait for data to appear on the queue?
- How can we wait for a queue to become empty? How do we wait for a consumer to finish processing everything in its queue? What's the difference?
Preliminaries
Synchronization
In a multi-threaded environment it is imperative that we use synchronization mechanisms to avoid situations like the producer trying to add something on the queue while simultaneously the consumer tries to get something from the queue. Without synchronization mechanisms this can lead to crashes or data corruption.Technology choices
To implement multi-threaded things in C++ in a portable way I have chosen the following technologies:- Boost: a C++ library that turns "C++" into "C++, batteries included"
- CMake: a cross-platform C++ build system
CMakeLists.txt
Here's my CMakeLists.txt file (New BSD License). This file defines how CMake can find the boost libraries, and build the project.
project(ProducerConsumer)
cmake_minimum_required(VERSION 2.8)
set(Boost_MULTITHREADED TRUE)
find_package (Boost COMPONENTS system thread)
include_directories (${Boost_INCLUDE_DIRS})
link_directories (${Boost_LIBRARY_DIRS})
aux_source_directory(. SRC_LIST)
add_executable(${PROJECT_NAME} ${SRC_LIST})
target_link_libraries(${PROJECT_NAME} ${Boost_LIBRARIES} )
To use this CMakeLists.txt file, you'd create a directory(say at: /home/user/myproject/src or c:\development\myproject\src) containing the following files- CMakeLists.txt (shown above)
- main.cpp (i.e. the rest of the code appearing later in this blog entry)
If you have gnu make installed:If you don't have gnu make installed on your system, try cmake --help to see how you can use it to generate a project file for eclipse, kdevelop, visual studio, ninja, codeblocks, ...
cmake ../src make
Creating a producer and consumer for testing
Here's a simple producer and consumer, both derived from a WorkerThread class (both need to run independently in a separate thread). WorkerThread implements all the things that all producers and consumers have in common, but leaves the actual work to be implemented by derived classes by means of the purely virtual method Work().In an attempt to reduce code duplication, I've templatized the WorkerThread with a QueueType that itself is templatized with QueueContents (e.g. int or some class PrintingParameters). In the rest of this blog entry, different QueueTypes will be presented.
Attempt 1: ThreadSafeQueue
Code
This is based on the code published here.Test program
Output
About to construct queue About to construct producer About to construct consumer About to start producer About to start consumer Waiting for producer to finish [PRODUCER] Produced value 0 [CONSUMER] Consumed value 0 [PRODUCER] Produced value 1 [PRODUCER] Produced value 2 [PRODUCER] Produced value 3 [PRODUCER] Produced value 4 [PRODUCER] Produced value 5 [PRODUCER] Produced value 6 [PRODUCER] Produced value 7 [PRODUCER] Produced value 8 [PRODUCER] Produced value 9 Waiting for consumer to finish [CONSUMER] Consumption of value 0 completely handled. [CONSUMER] Consumed value 1 [CONSUMER] Consumption of value 1 completely handled. [CONSUMER] Consumed value 2 [CONSUMER] Consumption of value 2 completely handled. [CONSUMER] Consumed value 3 [CONSUMER] Consumption of value 3 completely handled. [CONSUMER] Consumed value 4 [CONSUMER] Consumption of value 4 completely handled. [CONSUMER] Consumed value 5 [CONSUMER] Consumption of value 5 completely handled. [CONSUMER] Consumed value 6 [CONSUMER] Consumption of value 6 completely handled. [CONSUMER] Consumed value 7 [CONSUMER] Consumption of value 7 completely handled. [CONSUMER] Consumed value 8 [CONSUMER] Consumption of value 8 completely handled. [CONSUMER] Consumed value 9 [CONSUMER] Consumption of value 9 completely handled. Queue should be empty after all threads finished: 1
Discussion
- Each method accessing the queue first waits until it can acquire a mutex (obviously, each method has to try to acquire the same mutex). This is needed to ensure that other threads cannot modify the queue this method is operating on. This could lead to data corruption or crashes.
- At some point (e.g. in method TryPop) we check the condition "m_queue.size()==0". If we'd replace this condition with a call to "bool Empty()", the code would deadlock. The reason is that "Empty()" tries to acquire the same mutex that "TryPop(...)" has already acquired.
- To wait for data to appear on the queue, we wait for a condition variable. This is by far the most efficient way to wait for something to happen. In some code bases one finds constructs like "while (!DataAvailable) sleep(100);". Such approach consumes more resources and if one is not careful it is not thread-safe.
- When the producer is faster than the consumer (which I simulated in the test program by giving the consumer a 1 ms delay, and giving the producer no delay), it will keep on pushing values on the queue, and the queue size will keep increasing. This is clearly visible in the output of the program. At some point the available memory on the system would be exhausted causing a hang up or crash. This is a serious problem that needs an improved design: a BoundedThreadSafeQueue. A BoundedThreadSafeQueue is parameterized with a maximum size. Pushing data on the queue will block until the queue has room for more entries.
Attempt 2: BoundedThreadSafeQueue
Code
Test program
Output
About to construct queue About to construct producer About to construct consumer About to start producer About to start consumer Waiting for producer to finish Produced value 0 Produced value 1 Produced value 2 Consumed value 0 Produced value 3 Consumed value 1 Produced value 4 Consumed value 2 Produced value 5 Consumed value 3 Produced value 6 Consumed value 4 Produced value 7 Consumed value 5 Produced value 8 Consumed value 6 Produced value 9 Waiting for consumer to finish Consumed value 7 Consumed value 8 Consumed value 9 Queue should be empty after all threads finished: 1
Discussion
- The test program instantiated a bounded queue with size 3, with a consumer that is slower than the producer. The output of the test program shows that the consumer first needs to consume items before pushing can proceed when the queue is full.
- But we have a different problem now. In real life, consumption of an item will take some time (printing a large document can take several minutes). The queue can now signal that it is empty, but an empty queue does not mean that all processing is finished. (The items are removed from the queue before their processing starts.) So how can we extend our design to be able to wait until the queue and associated processing is completely finished?
Attempt 3: BoundedThreadSafeQueueSignalWorkloadDone
The basic idea is for the consumer of the queue data to inform the queue that processing the data is finished. Ideally we'd have a kind of condition variable that we can set/reset manually to signal that the queue + associated processing is completely finished (idle) or not. Someone waiting for the queue and associated processing to finish completely can then wait for the condition variable to become true. Boost doesn't offer this out of the box, but based on the code published here, it is possible to create such a manually resetable condition variable ourselves.
The consumer class needs a small update: we have to call ProcessingFinished() on the queue after finishing processing the information.
Output
[MAIN] About to construct queue [MAIN] About to construct producer [MAIN] About to construct consumer [MAIN] About to start producer [MAIN] About to start consumer [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [PRODUCER] Produced value 0 [CONSUMER] Consumed value 0 [PRODUCER] Produced value 1 [PRODUCER] Produced value 2 [PRODUCER] Produced value 3 [PRODUCER] Produced value 4 [QUEUE] Wait until idle [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 0 completely handled. [CONSUMER] Consumed value 1 [PRODUCER] Produced value 5 [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 1 completely handled. [CONSUMER] Consumed value 2 [PRODUCER] Produced value 6 [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 2 completely handled. [CONSUMER] Consumed value 3 [PRODUCER] Produced value 7 [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 3 completely handled. [CONSUMER] Consumed value 4 [PRODUCER] Produced value 8 [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 4 completely handled. [CONSUMER] Consumed value 5 [PRODUCER] Produced value 9 [QUEUE] ProcessingFinished but queue not empty. ProcessingDone flag remains low. [CONSUMER] Consumption of value 5 completely handled. [CONSUMER] Consumed value 6 [QUEUE] ProcessingFinished but queue not empty [CONSUMER] Consumption of value 6 completely handled. [CONSUMER] Consumed value 7 [QUEUE] ProcessingFinished but queue not empty [CONSUMER] Consumption of value 7 completely handled. [CONSUMER] Consumed value 8 [QUEUE] ProcessingFinished but queue not empty [CONSUMER] Consumption of value 8 completely handled. [CONSUMER] Consumed value 9 [CONSUMER] Consumption of value 9 completely handled. [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [QUEUE] Wait until idle [QUEUE] Consumer is completely idle [MAIN] Queue should be empty after all threads finished: 1 [MAIN] Waiting for producer to finish [MAIN] Waiting for consumer to finish
Discussion
- Writing a meaningful test program is less obvious here.
- From the output: when the program starts, it starts producer and consumer in a separate thread, and then starts waiting in the main thread until the queue + associated processing is completely finished.
- Just after constructing the producer and consumer, the channel is still completely idle: no work has been pushed onto the queue yet.
- Then the producer starts producing values, and the next wait until idle blocks until the processing is completely finished. After that, WaitUntilCompletelyIdle immediately returns true (until another producer inserts work into the queue).
Friday, June 1, 2012
Learning to read music notes faster
The Problem
I love to learn new music, but I experience difficulty reading notes that lie high above or below the staff fast enough. Being into programming I thought I'd create something to help me, and while doing so learned some neat things.Requirements
- I imagine a kind of flash card system: a program that displays random music notes, lets me guess the note names and then tells me how long it took to finish all the assignments as a simple way of tracking my progress in reading notes faster.
- I want to be able to configure which notes I'm questioned about: one day I may want to practice low notes, another day I want to practice high notes. And I can imagine beginning musicians would be happy to practice "easier" notes as well.
- The system should support at least the g-clef and the f-clef (the two most commonly used clefs).
- To quickly reach some result, I want to prerender all the graphics. This also allows me to use software to generate professional looking graphics. The icons that go on the buttons must be automatically generated too, as I'm not interested in manually creating and maintaining icons.
- Finally, I want the program to run on at least linux and windows systems. I don't have a Mac to test on, but preferably it should be easily portable to Mac's too.
Technology choices
- The program is written in python 2.x and pyqt4. The code is available under GPLv3 license. To create the graphics
- I first programmatically generate lilypond files.
- I then render those .ly files to scalable vector graphics (svg) format. Display of the assignment can immediately use the .svg format. SVG being a vector format, this will ensure maximum quality.
- The icons on the buttons to the best of my knowledge must be in bitmap format. First, I render the SVG files to portable network graphics (.png) using inkscape. Inkscape provides some convenient command line options that allow just that. Inkscape allows me to export the SVG bounding box to a .png.
- I wanted to have a very general way to do postprocessing on the icons, and found Imagemagick. Imagemagick is like a swiss knife of bitmap processing, and best of all it allows to specify all the operations programmatically. I know to some of you this doesn't sound very exciting, but that's usually because you don't fully realize what incredible possibilities Imagemagick has. It's quite hard to overestimate the awesomeness of Imagemagick! I use Imagemagick to automatically make the icon size 64x64, with the graphics nicely centered. (That is to say: the icons are only displayed at 48x48 right now, with the resizing done in Qt - I may revise these settings in the future. The nice thing about having everything automated is that it becomes easy to make experiments with different settings).
- The build system: GNU make (for now). As I've used it before it seemed like the quickest way to get something. In retrospect, I probably should look out for something else. As I found out the hard way, GNU make has some serious trouble when it comes to dealing with filenames containing whitespace. It also is less than ideal for cross-platform development. Sure, there's a native win32 version of GNU make available, but one of the things bothers me is how its internal variable $(RM) on windows platforms still is defined as
rm -f
, making it harder than needed to implement amake clean
on windows.
The code
The code can be found on http://speedsightread.sourceforge.net/. It also contains an .nsis file to generate a nullsoft installshield (setup.exe) for windows systems. The make clean target for now only works on linux systems. There are no pre-built versions available yet.
Building the code
- To build the program you need: python 2.x (I used python 2.7), pyqt4, lilypond, imagemagick, GNU make
- To build an installshield you additionally need: py2exe, nullsoft nsis
- To run the program, once it's built, you only need python 2.x and pyqt4. If you build the windows setup.exe using the nullsoft nsis script, you don't need anything else but the setup.exe.
Current status
As usual with python based software, I got lots of result for a relatively small effort. It-works-for-me. No pre-built versions available yet. It remains to be seen if I can actually improve my sight reading skills using it :)A final screenshot
Here's another screenshot, running on linux (icewm with an XP theme):
Friday, May 4, 2012
Pyjamas hijacked!
What is pyjamas?
Pyjamas is a port to python of google's web-toolkit (GWT). Pyjamas allows one to write python code, and have it automatically translated into javascript. The generated javascript then can be run inside a browser as a web application, or can be run unchanged in a suitable run-time environment as a desktop application. In both cases, the applications can communicate with web servers via AJAX calls.What happened?
During a heated discussion about what it would take to promote pyjamas to a larger audience, some people got angry with pyjama's main developer and innovator, Luke Leighton. In summary, a few people were not happy with the existing infrastructure used by the project whereas the main developer rather bluntly dismissed the concerns and told people to concentrate on things that really matter instead. Furthermore, Luke told the people that pyjamas is bloody innovative. innovations are disruptive. get used to innovation being disruptive!. . People got really upset and concocted some sort of master plan to prove the world once and for all they were not to be laughed with. Innovation disruptive? They were going to teach Luke about innovation!The normal procedure when people are not happy with the way an open source project is lead, is to create a fork and let both parties go their own way. This is more or less what happened here too, but with an evil twist (read on, stuff like this doesn't happen every day).
As it turns out Luke Leighton didn't hold the rights to the domain name pyjs.org, and the disgruntled people (at least one of which had admin rights on Luke's server to "help out" with administering it) contacted the rightful owner of the domain, and convinced him to transfer the domain name to the dissidents. Then, suddenly and out of the blue came an [[ANNOUNCEMENT]] on the (now defunct) pyjamas-devel mailing list (although to be honest there had been a vague hint to it in the past). In the announcement it was said how pyjs.org from now on points to a new location, where the pyjamas source code and wiki can be found on github, and the mailing list has been replaced with mailman3 software, and it was pointed out that Luke Leighton from now on was no longer in charge of the project in its new incarnation. Until today it's not entirely clear if private data has been copied from Luke's server to recreate the mailing list using new software. If that is indeed the case, it might consist of a criminal act (data theft) in at least one of the involved countries, and we may not yet have seen the last of this.
Effects of the hijacking
Well, the basic aim of the disgruntled people is to have better infrastructure, to make the project look more mature and attract more developers. How well did they succeed in achieving their goals with their actions? The current state of things for an outsider, at the time of writing 4 days after the facts, is not looking very good to say the least (although the dissidents, I'm sure, will disagree):- The new and "improved" website lacks basic functionality: most links are broken; the manual is offline; the examples are no longer accessible. Most links in google's archives now point to non-existing pages. Contact links still point to the old defunct mailing list, etc. I assume this will be solved over time, but in a proper transition to new infrastructure, this should have been solved before transitioning. Someone clearly was in a lot of haste to pull off this underhand trick, and apparently all this haste lead to getting very tired, too tired to fix the basic functionality. (And oh the irony of fate! While I'm writing this post, the google groups appear to suffer from very slow reply times. Seems like the shiny new technology has its share of flaws as well...)
- During the recreation of the mailing list under the new ownership with new mailing list software, something went wrong and tens of people who had registered to the mailing list, but had asked not to receive emails (e.g. because they follow the group using an nntp reader via gmane) now suddenly got flooded with emails from the pyjamas list and started sending mails to unsubscribe (which were then received by all the other people who also didn't want to receive emails from the mailing list, who then also started to send unsubscribe mails...) The unsubscription emails didn't work, because the new mailing list software expects different commands (but of course no one had been informed about that), which led to an avalanche of "please unsubscribe me" and "please stop unsubscribing" messages, and eventually to killing off the mailing list, and creating a new one here. Looking at the number of people who've subscribed to the new list so far (5 at the time of writing, one of which already made it clear that he doesn't agree to how the changes had been hastily enforced), it would seem that this whole operation is not exactly something a 600+ community of people was waiting for.
- Since the hijacking, only a few individuals have publicly applauded the event. So far, the hijack seems to have happened to serve the interests of those few individuals. No discussion about the matter happened on the mailing lists. No wonder a large part of the existing community is shocked and, frankly, pissed off (including, it seems, many of the current top committers). No matter how many professional tools one uses, without active committers the future of the project looks very uncertain. And with the main developer and innovator basically having been stabbed in the back, it can be expected that he won't be contributing to the project anymore (in the best case he would continue in a fork of his own, perhaps under a new name, but at this moment nothing's certain).
- Corporate users of the framework now consider the whole project to be avoided-at-all-cost. No one in his right mind wants to base a business on a project hijacked by a minority of impulsive hotheads, without clear guarantees for continued development and support.
Conclusion
Even if the project manages to survive and attract new developers (and I really hope it does in one way or another, because pyjamas at its core is really cool technology!) the current state of matters where the main contributor and driving force Luke Leighton is as-good-as-eliminated from the project, where the fully functioning website is hastily replaced with something completely broken for days in a row and while making a significant part of the active and passive users unhappy is probably the worst possible thing that could have happened to it, and despite the new and shiny tools, the project might take a long time to reach a decent velocity, and gain some credibility with corporate users again.Sunday, April 1, 2012
Mutable state considered harmful?
Image by Török Gábor (nyuhuhuu) on FlickrMusings on mutable state
In this video, Misko Hevery argues that global state is bad from a testability point of view. Global state resides in global variables and singletons. They make the API deceptive in the sense that when you repeat the same calls, you can still expect different results because the results depend on both the object's state and on global state which is not obvious to someone reading the code. After seeing that video it occurred to me that the exact same argument applies to mutable data members in a class. They are a kind of "globally available state inside a given object instance" that can make the API deceptive. Consider the following code: As you can see the "Display" method changes internal state of a TestClass instance, and causes different results for every new API invocation. So the arguments Misko Hevery use seem valid ones, but applying them rigorously has deep implications: it seems to me it really argues against side-effects and thus in favor of purely functional programming. By now I have seen a fair share of hard-to-trace bugs caused by side-effects of method calls, so it struck me that perhaps it is possible to find some compromise between mutable state and purely function programming (and it's really just that: an idea resulting from a brainstorm; I haven't actually used it in production as I fear it may have significant performance implications). If you have additional insights into the matter, feel free to comment.Proposal: Forbid non-const methods.
How is this different from existing practices?
Difference with const correctness
Const correctness argues that one should prefer const methods and passing objects by references/pointers to const, but it doesn't forbid non-const methods. In my proposal, non-const methods are forbidden (e.g. enforced by a suitable compiler). Note that in my proposal it is not forbidden to pass references to non-const objects as arguments. So in my proposal, objects cannot change themselves but they can change arguments they receive (in practice, this means local variables of plain old data types, as other objects only have const methods which prevents them from being changed). The fact that you have to pass in the arguments as function parameters makes it obvious that the code might change them. This is a case of changing state that doesn't happen "behind the scenes", and therefore can be defended.Difference with purely functional programming
In my proposal it is still allowed to have mutable local variables of plain old data types, which allows writing "for"-loops without having to resort to recursion. I have nothing against recursion per se, and in the proper context it can make code more concise and readable, but in many cases for-loops are simpler to understand, and more efficient in terms of run-time and memory usage (especially in languages like C++ which do not have built-in tail call elimination). Interestingly, the side-effects in constructors and destructors are what make tail call elimination very hard to impossible in general in C++ -like languages.What can you do with immutable objects? Isn't the very point of defining an object to group together the data and the operations that operate on that data?
An object indeed groups data and operations, but nothing prevents the object from creating a new object as a result of performing operations on its own data (as opposed to modifying its own internal state). Instead of modifying objects, you can always create a new object with the desired values. The old object can either be thrown away, or kept around in case someone still needs it.But isn't this incredibly inefficient from a performance point of view?
On the one hand, there probably is some price to pay. On the other hand, one probably could fine-tune the run-time environment to make creation of many similar objects cheaper (perhaps by keeping around and reusing discarded objects?). Add to that the fact that modern computers are fast enough to deliver reasonable performance with much more exotic ideas like dynamic typing. Let's not forget why we would use this system: to limit the impact of side-effects, and have less trouble understanding how our code works. The advantage, should it work the way it is envisioned, would be in more readable code, a reduced bug count and easier to trace bugs and therefore increased programmer productivity.Saturday, March 24, 2012
Syntax highlighting stax scripts in vim
What is Staf/Stax?
STAF is a Software Testing Automation Framework. It provides all basic infrastructure for creating robust distributed test environments. It's structured as a collection of services which can be called from the command line or from a ui, or from your favourite programming language. These services implement things like: "start a process on pc xxx and get the return code", "copy files from pc xxx to pc yyy", synchronization primitives, sharing of variables between pcs, sending mails, etc etc etc. STAF is a very impressive piece of software. Don't let the outdated JAVA ui look and feel of some of the tools that come with it fool you. Everything can be activated from the command line. Making use of the JAVA uis is optional, but the ui offers lots of out-of-the-box functionality that I wouldn't want to program myself if I can avoid it.Syntax highlighting code
Put the following code in a file called stax.vim:runtime! syntax/xml.vim unlet! b:current_syntax syntax include @Python syntax/python.vim syntax region pythoncode matchgroup=pythongroup start=+<script>+ keepend end=+</script>+ contains=@Python hi link pythongroup xmlTagOn my linux system, in the .vimrc file, I added
au BufNewFile,BufRead *.stax set filetype=staxto activate the syntax highlighting for all files that have a .stax file extension. On a windows system, i had to add the above line to "Program Files"\Vim\vim73\filetype.vim instead. With the syntax highlighting definition provided here, STAX becomes more programmer friendly. (Note: I posted this same code some time ago on the staf/stax mailing list)
Thursday, January 5, 2012
Capture colored console output in a tkinter window
The problem
I want to run a console script that emits ansi color codes to generate a colorful output and capture that colorful output in a tkinter window. Ideally I want this to work for programs like cmake or emerge which semi-intelligently switch off color codes if they detect they are not running on a terminal. This is all tested under linux. Some if it might work on windows too, but I haven't investigated.Approach
I will show some increasingly sophisticated approaches using python 3.2 and try to outline their limits or problems. In order not to overload the presentation, this section details some functions that are used by the different approaches below. First of all in a file ansicolortext.py I have the following MIT licensed code. It is a tkinter Text widget that tranforms ansi color code sequences into tkinter color tags. Next, in a file timeout.py I have the following code, created by Matt Harrison under MIT license and posted as an ActiveState recipe: Finally, here's a function to create the user interface (a tkinter window with a resizable AnsiColorText widget)Approach 1
If you google for this type of stuff almost always someone will come up with something as follows: This simple approach works beautifully for the example given (ls --color) and is highly recommended if it works for you. For some programs, however, this doesn't work. Many programs try to detect if they are running on a terminal or if they are part of a piped series of commands, and in the latter case suppress color codes. Examples of such programs include CMake and grep with the --color option. The end result is that your output shows up in black and white.Approach 2, attempt 1
In order to fool programs like CMake or grep into producing color codes we need to pretend we are a terminal. The way to do this in linux is by using a pseudoterminal. According to wikipedia,In some operating systems, including Unix, a pseudo terminal is a pseudo-device pair that provides a text terminal interface without an associated device, such as a virtual console, computer terminal or serial port. Instead, a process assumes the role of the underlying hardware for the pseudo terminal session.Pseudoterminals can be accessed using Python's pty module. pty.openpty() opens a new pseudo-terminal pair and returns a pair of file descriptors (master, slave), for the master and the slave end, respectively. The terminal emulator process is associated with the master device and the shell is associated with the slave. The terminal emulator process (master) receives input from the keyboard and mouse using windowing events, and is thus able to transmit these characters to the shell (slave), giving the shell the appearance of the terminal emulator being an underlying hardware object. Any terminal operations performed by the shell in a terminal emulator session are received and handled by the terminal emulator process itself (such as terminal resizing or terminal resets). This approach looks clean and simple, but it has a problem: os.read blocks when no more data comes in, and so our script hangs before the UI can even come up. One proposal could be to use a timeout when reading data using os.read.
Approach 2, attempt 2
This approach now deals with the blocking: upon a timeout the communication is stopped. It suffers from a different problem though: the UI doesn't appear before all data has come in. In the case of a CMake script this can take minutes to hours to complete. Not exactly what we wanted... therefore we need a slightly smarter approach.Approach 2, attempt 3
The idea is this: we will run all the communication in a thread of its own. The UI will run in the main thread. The communication thread will send the data it receives to the main thread by means of a Queue object. The main thread will read data from the Queue and display it in the text widget as it receives it.The TextUpdater object T lives in the program's main thread. It reads data from the communication thread t by means of Queue q and sends it to the Text widget text, which itself is a child widget of the Tk root window root. The TextUpdater thread is asked to start working 100ms after root.mainloop() is started. Like that the UI has ample time to get itself ready for processing. The communication thread ThreadCommand t opens a pseudoterminal and uses it to communicate with the program being run. It stops itself and flags itself as not running anymore if no more data comes from the process (i.e. if the reading timed out). Reading data from the process uses a version of os.read with a timeout added to it (read2). The data read by read2 is slightly processed before it is sent to the communication thread: only complete lines of text are sent to avoid that we break up text in the middle of a color code (this would give trouble because the AnsiColorText widget isn't smart enough to deal with half-finished color codes).
If the timeout value used in read2 is too small, the timeout can occur too soon (especially with processes that take some time to generate output, like CMake) so this part is still a weak spot in this approach. Too long a timeout causes one to wait long after the process finishes to really close the communication thread. If you know of better ways to handle this, please comment! After text is written to the Text widget, we call root.update() to give Tk() the chance to handle pending events. This makes sure that the text widget is updated as input comes in, and it also makes sure that you can resize the window or scroll through the displayed data while data is still coming in.
An alternative without timeout decorator
As I found out today, there's an alternative without timeout decorater, by using the poll call of Python's select module. It still uses a timeout mechanism to decide if there's data to be read. This time the timeout mechanism is provided by the select module directly.Screenshot
Here's a screenshot of CMake producing colored output in a Tk window. While CMake is running, the window can be resized and you can navigate through it with arrow/pageUp/pageDown/... keys.
Subscribe to:
Posts (Atom)