Driving the GODOT game engine with OSC messages

Problem?

Note: I've tested this code with a very recent version of GODOT compiled from source code. Your mileage may vary.

In interactive art installations, an often used protocol to communicate between computer processes is the open sound control (OSC) protocol. Despite the name "sound" in OSC, the protocol is actually completely general purpose. In my opinion a game engine like the godot engine would benefit tremendously from built-in support for the OSC protocol because it opens up the engine to creative uses outside the gaming domain. On of my own use cases is to visualize in real-time musical events that are algorithmically generated (also in real-time) in supercollider.

The GODOT development team, despite several requests from users, chose not to support OSC directly and provide a high-level networking interface which they feel is more appropriate for their game engine (but which only works with their game engine). 

As a result, some users have attempted to write add-on modules using the GdNative C++ add-on mechanism to extend the GODOT engine. Such modules often have the drawback of limited portability and requiring users to set up a C++ building environment. In addition, the API that needs to be used by the add-ons could be changing drastically between GODOT versions. 

Therefore, I was hoping to find a more portable solution. Luckily GODOT also provides some lower level networking classes, which allow third-party developers to add some support for unsupported protocols.

OSC is typically a thin layer on top of UDP networking (sometimes TCP is used, but most often UDP is chosen for reasons of performance), and so it should be possible to add a simple OSC implementation in GDScript directly.

Note: to make sense of this blog post, you need to have a little bit of working knowledge about the GODOT game engine, at the minimum, you need to know what nodes are.

Will it perform super fast? 

Probably not. But often you don't really need to send tons of data over the network to drive your visualization. Think about optimizing your data streams.

Will it be easy to use?

I think it is extremely easy to use. But feel free to try it out and judge for yourself.

How can I use it?

Glad you ask. Here's the big picture:

To the root note of our GODOT sketch (usually a Node2D for 2d games, or a Spatial for 3d games), I attach a script that monitors the network for incoming OSC messages. The OSC messages are decoded and automatically dispatched to child nodes who requested to be kept informed about certain incoming messages.

Consider an example of a Node2D rootnode (which I renamed to RootNode in the IDE) with a Sprite child node. In the script attached to the Sprite, I only have to express my desire to be kept up-to-date about incoming OSC messages, e.g. here's the complete script for a sprite that can change position by receiving a /set/xy OSC message.

extends Sprite

func _ready():
   get_tree().get_root().get_node("RootNode").register_dual_arg_callback("/set/xy", get_node("."), "set_xy")

func set_xy(x, y):
   self.position.x = x
   self.position.y = y

This means that when the Sprite is instantiated, it registers itself with the root node ("register_dual_arg_callback"). It tells the root note that it wants its function "set_xy" to be called whenever a "/set/xy" OSC message is received over the network with two arguments (x and y) which in this example then are used to set the sprite position. There's also a register_single_arg_callback for functions that expect only one value (and nothing prevents you from adding more variants).

This simple code suffices to ensure that from now on, this particular Sprite will automatically react to incoming OSC /set/xy messages by updating its position.

E.g. in an environment that supports sending OSC message like supercollider, I can run the following snippet to update the sprite position:

(

b = NetAddr.new("127.0.0.1", 4242); // create the NetAddr
b.sendMsg("/set/xy", 650, 200);

)

The godot sketch automatically reacts to the OSC message by updating the sprite xy position to 650, 200, exactly what we requested the system to do by registering a callback. Needless to say the OSC message could perfectly come from another computer, tablet or phone (e.g. from the TouchOsc app or from Open Stage Control).

The value 4242 is the port number on which the root node is listening for incoming OSC messages. In the proof of concept code it's hardcoded to 4242, but you are of course free to change it or to make it configurable.

Ahm, ok. Show me this magic code that goes into the root node then?

Note: the code in this section is obsolete, and replaced by the code in part 2 of this article: https://technogems.blogspot.com/2021/11/driving-godot-game-engine-with-osc.html

Well... it's not the prettiest code (it's a proof of concept - things like IP address and port are hardcoded, but should be simple to change). It works well enough for me at the moment (but I've only done some basic experiments so far). In case I run into trouble, I may need to revisit the details. If you run into trouble, please let me know what happened and how you solved it (or just explain the problem and we can discuss to see if we can find a solution).

Here's the code I put in the root node. For now it only supports parsing integer, float, string and blob OSC messages. In the future maybe also OSC bundles and some other data types could be added, but even with only this simple subset of OSC supported, the possibilities are already endless.

Note that the call to OS.set_low_processor_usage_mode may not be portable across all platforms, but it's probably not strictly needed to make the system work (but don't take my word for it, I'm not at all experienced with GODOT on platforms other than linux).

The code for registering callbacks and dispatching probably can be made a bit more general, and there could be support for pausing, resuming and stopping reacting to OSC notifications but for demonstration purposes what I have here should suffice.

extends Node2D

var IP_CLIENT
var PORT_CLIENT
var PORT_SERVER = 4242 # change me if you like!
var IP_SERVER = "127.0.0.1" # change me if you like!
var socketUDP = PacketPeerUDP.new()
var observers = Dictionary()

func register_single_arg_callback(oscaddress, node, functionname):
 observers[oscaddress] = [1, node, functionname]
 
func register_dual_arg_callback(oscaddress, node, functionname):
 observers[oscaddress] = [2, node, functionname]
 
func _ready():
 OS.set_low_processor_usage_mode(true)
 start_server()
 
func all_zeros(lst):
 if lst == []:
  return true
 for el in lst:
  if el != 0:
   return false
 return true

func _process(_delta):
 if socketUDP.get_available_packet_count() > 0:
  var array_bytes = socketUDP.get_packet()
  #var IP_CLIENT = socketUDP.get_packet_ip()
  #var PORT_CLIENT = socketUDP.get_packet_port()
  var stream = StreamPeerBuffer.new()
  stream.set_data_array(array_bytes)
  stream.set_big_endian(true)
  var address_finished = false
  var type_finished = false
  var address = ""
  var type = ""

                # parse osc address
  while not address_finished:
   for _i in range(4):
    var addrpart = stream.get_u8()
    if addrpart != 0:
     address += char(addrpart)
    if addrpart == 0:
     address_finished = true
    

                # parse osc type list 
  while not type_finished:
   for _i in range(4):
    var c = stream.get_u8()
    if c != 0 and char(c) != ",":
     type += char(c)
    if c == 0:
     type_finished = true

  # decode values from the stream
  var values = []
  for type_id in type:
   if type_id == "i":
    var intval = stream.get_32()
    values.append(intval)
   elif type_id == "f":
    var floatval = stream.get_float()
    values.append(floatval)
   elif type_id == "s":
    var stringval = ""
    var string_finished = false
    while not string_finished:
     for _i in range(4):
      var ch = stream.get_u8()
      if ch != 0:
       stringval += char(ch)
      else:
       string_finished = true
    values.append(stringval)
   elif type_id == "b":
    var data = []
    var count = stream.get_u32()
    var idx = 0
    var blob_finished = false
    while not blob_finished:
     for _i in range(4):
      var ch = stream.get_u8()
      if idx < count:
       data.append(ch)
      idx += 1
      if idx >= count:
       blob_finished = true
    values.append(data)
   else:
    printt("type " + type_id +" not yet supported")

  if observers.has(address):
   var observer = observers[address]
   var number_args = observer[0]
   var nodepath = observer[1]
   var funcname = observer[2]
   if number_args == 1:
    nodepath.call(funcname, values[0])
   elif number_args == 2:
    nodepath.call(funcname, values[0], values[1])
    
func start_server():
 if (socketUDP.listen(PORT_SERVER) != OK):
  printt("Error listening on port: " + str(PORT_SERVER))
 else:
  printt("Listening on port: " + str(PORT_SERVER))

func _exit_tree():
 socketUDP.close()

Compared to the misery of adding, compiling, maintaining, porting, ... a GdNative module, I think this is pretty acceptable (at least for my use cases).

Making an arpeggiator in supercollider with patterns

Problem

Given some notes as input, generate a pattern making use of those notes. When the input changes, the generated pattern should also change. Most keyboards and synthesizers provide simple arpeggiators, but we'll be using supercollider which allows for generating the most complex patterns imaginable including generation of chords, polyphony, introducing random variations, etc
Code for the final piece of code in this post can be found on sccode.org: http://sccode.org/1-5cr

Approach

How can we convert a list of notes to an interesting arpeggio? Different possibilities exist, but we'll be using one of the more powerful abstractions available to supercollider users: the pattern system.

Patterns act as a kind of template for generating events, and events can be played to create sounds. This sounds exactly like what the doctor prescribed.

A simple example to get started

First let's assign some midi notes to a variable ~n:

~n = [60, 64, 67]; // c major chord

If you, like me, prefer to reason in note names instead, you can install the Panola quark:

Quarks.install("https://github.com/shimpe/panola");

Then you can write the following instead (it's a bit longer but, hey,  at least with readable note names):

~n = Panola("c4 e4 g4").midinotePattern.asStream.all;

Our task is to define a pattern that uses the notes in ~n and builds from them a simple arpeggio. So, given the input notes [c4, e4, g4] instead of playing simply the notes [c4, e4, g4] we'll generate a more interesting arpeggio [c4, g4, e4, g4]. When the input notes change to e.g. [c4, f4, a4] the arpeggio that is generated should change to [c4, a4, f4, a4].

(
s.waitForBoot({
    var arp = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([n0, n2, n1, n2])
        },
        \dur, Pseq([1,1,1,1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(arp).play;
});
)

This code requires some explanation:

• s.waitForBoot is a construct I use in pretty much every supercollider sketch I make. It will start the supercollider sound server if it wasn't started yet.
• Once the server is booted, the function that is passed to waitForBoot is executed. This function defines a pattern (also known as Pbind) "arp" and plays it.
• The full power of Pbind is available (meaning that you could e.g. generate midi events and send them to a hardware instrument), but for demo purposes we just instantiate the supercollider default instrument. This should generate some sound even if you don't own any fancy hardware (after all, supercollider is also a sound synthesis language and therefore very capable of generating its own sounds). Instantiating the default instrument is accomplished by specifying the key-value pair \instrument, \default in the Pbind.

Plazy is a filter pattern that allows to calculate a new supercollider pattern using a function.

In addition to selecting an instrument, we also need to generate the notes to be played. Remember that ~n is the input note list. First we try to extract the first 3 notes from the input note list. These are the notes that we will use to rearrange into our arpeggio.

The line ~n = ~n ?? [Rest(1)] checks if variable ~n is defined already (actually it checks if ~n is nil). If it is not defined, it is assigned a list of input notes consisting of a single Rest. Then I introduce some variables n0, n1, n2 to denote the first, second and third note in the input list respectively. It may happen that the input list contains fewer than 3 notes (e.g. if you initialize the ~n variable from midi input from a hardware device, someone might play a 2-note chord instead of a 3-note chord). In that case we don't want our code to crash. If the first note is missing, I replace it with a Rest. If the second or third notes are missing, they are replaced with the first note.

The function passed to Plazy returns a Pseq that generates our arpeggio consisting of the first, third, second and third input note: n0, n2, n1, n2. Pseq is a pattern that generates successive notes. By default the complete list of notes will be repeated once and then the pattern stops.

To generate an arpeggio we are not limited to only generating notes. We can also generate durations, volumes, legato-staccato, and a bunch of other properties. All these can be derived from the input note list, or they can be completely independent from them if so desired. In this first example, let's just give all notes equal duration and keep all other properties to their default values. Specifying durations in a pattern is done by using the \dur key.

I want the complete arpeggio to be finished in 2 beats, so I specify 4 (because there are 4 notes) equal relative durations of 1 and then normalize the values to make their sum == 2.

Finally, we need to stop any previous instances of the pattern that may be playing, and make our pattern start. This happens in the lines

if (~player.notNil) { ~player.stop; }; // call stop if not stopped already
~player = Pn(arp).play;

Note the use of the Pn pattern, to make our arpeggiator repeat indefinitely.

As soon as you redefine the ~n variable to a new value, the pattern (but only after the previous instance was completely finished) will use the new values in ~n and generate a new arpeggio built from the new notes, so e.g. while the pattern is playing, try evaluating the following lines one by one, listening how it changes the arpeggio.

~n = [60, 64, 67 ];

~n =[ 60, 65, 69 ];

~n =[ 59, 65, 67 ]; 

Exercise: adapt the code to make an arpeggio based on the first four notes of a list of input notes.

Exercise: adapt the code to use different durations for different notes

Fancier patterns

The simple arpeggio we generated above is already more complex than what many synthesizers and keyboards can do, but supercollider being supercollider this is just the tip of the iceberg available to us.

We can generate multiple patterns from a single list of input notes and play them together with Ppar. In addition, not all durations and amplitudes need to be the same. You could generate a complete 16 track auto-accompaniment from a simple list of input notes using this technique. Here's an example of a melody pattern with a bass line generated from the input notes. Note that I do some arithmetic on the notes (+12) to add an octave. In general you are not limited to using only the input notes given by the user. You can add any other note you desire, which can (but needn't) be derived from one of the notes in the input list.

(
s.waitForBoot({
    var right, left;
    ~n = ~n ?? [Rest(1)];
    right = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n1, n2 ] ++  (([ n0, n2, n1, n2 ] + 12)!2).flatten)
        },
        \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
    );
    left = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12)
        },
        \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(Ppar([right,left])).play;
});
)

Exercise: define some percussion instruments and add some percussion to the fragment.

Listening to midi input

Until now we've defined the ~n variable manually. But we can just well listen to a midi device and react to the incoming notes (or control change messages).
Before we can use midi devices, we need to initialize supercollider's midi system. To do so, evaluate the following code:

(
MIDIdef.freeAll;
MIDIClient.init;
MIDIIn.connectAll;
)

MIDIdef.freeAll will remove any midi handlers that may still be running. MIDIClient.init will initialize midi communication in supercollider and MIDIIn.connectAll ensures that we react to incoming midi msgs of all midi devices connected to the system.

Now we can install a midi handler that reacts to note on and note off messages. We will maintain an array of notes, and for each note in the array whether it's on or off. This array forms the basis from which we derive our list of input notes.

Note that in the following, the arpeggio keeps playing until we press another chord. If you want the arpeggio to stop when you release the midi keys, you can add

~n = ~note_table.selectIndices({|item, i| item != 0});
if (~n == []) { ~n = nil; };

in the note off handler.

(
MIDIdef.freeAll;
MIDIClient.init;
MIDIIn.connectAll;
)

(
s.waitForBoot({
    var right, left;

    ~note_table = 0!127;
    
    MIDIdef.noteOn(
        \mynoteonhandler, // just a name for this handler
        {
            |val, num, chan, src|
            num.debug("num");
            ~note_table[num] = 1; // update note table and update ~n
            ~n = ~note_table.selectIndices({|item, i| item != 0}).postln;
        }
    );

    MIDIdef.noteOff(
        \mynoteoffhandler, // just a name for this handler
        {
            |val, num, chan, src|
            num.debug("num");
            ~note_table[num] = 0; // update note table and update ~n
        }
    );

    right = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pxrand([ n0, n2, n1, n2 ] ++  (([ n0, n2, n1, n2 ] + 12)!2).flatten)
        },
        \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
    );
    left = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12)
        },
        \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(Ppar([right,left])).play;
});
)

Note that now always a complete "iteration" of the pattern has to be finished before changes in the input note list create a new arpeggio. You cannot switch the arpeggio to a different chord in the middle of the arpeggio. How can we change this behaviour?

Change chords in the middle of the arpeggio

What if we want changes in the input notes to have immediate effect? Can we adapt the code to make the system react faster to changes? Well... it's supercollider so of course we can. Let's see how it can be done.
If we want immediate reaction of the system to chord changes, one approach is to replace n0, n1, n2 with patterns that reevaluate a function every time they are called. This function then performs a lookup of a note in our ~n variable, which is updated as soon as new midi notes are received.
Also I moved the midi initialization code inside the system because I don't really like having to evaluate multiple code blocks successively.

(
s.waitForBoot({
 var right, left;
 var n0, n1, n2;

 MIDIdef.freeAll;
 if (~midi_initialized.isNil) {
  MIDIClient.init;
  MIDIIn.connectAll;
  ~midi_initialized = 1;
 };

 ~note_table = 0!127;
 ~n = nil;

 MIDIdef.noteOn(
  \mynoteonhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 1; // update note table and update ~n
   ~n = ~note_table.selectIndices({|item, i| item != 0});
  }
 );

 MIDIdef.noteOff(
  \mynoteoffhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 0; // update note table and update ~n
   /*
   // enable next two lines only if you want arpeggios to stop playing
   // when you release the midi keys
   ~n = ~note_table.selectIndices({|item, i| item != 0});
   if (~n == []) { ~n = nil; };
   */
  }
 );

 n0 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   ~n[0] ?? Pseq([Rest(1)]);
  };
 };

 n1 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   Pseq([~n[1] ?? ~n[0]]);
  };
 };

 n2 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   Pseq([~n[2] ?? ~n[0]]);
  };
 };

 right = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n1, n2] ++ (([ n0, n2, n1, n2] + 12)!2).flatten),
  \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
 );
 left = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12),
  \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
 );
 if (~player.notNil) { ~player.stop; };
 ~player = Pn(Ppar([right,left])).play;
});
)

Final cleanup

As a final cleanup we can remove some code duplication

(
s.waitForBoot({
 var right, left;
 var n0, n1, n2;
 var note_getter;

 MIDIdef.freeAll;
 if (~midi_initilized.isNil) {
  MIDIClient.init;
  MIDIIn.connectAll;
  ~midi_initialized = 1;
 };

 ~note_table = 0!127;
 ~n = nil;

 MIDIdef.noteOn(
  \mynoteonhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 1; // update note table and update ~n
   ~n = ~note_table.selectIndices({|item, i| item != 0});
  }
 );

 MIDIdef.noteOff(
  \mynoteoffhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 0; // update note table and update ~n
   /*
   // only enable the following lines if you want the arpeggio to stop as soon as you release the keys
   ~n = ~note_table.selectIndices({|item, i| item != 0});
   if (~n == []) { ~n = nil; };
   */
  }
 );

 note_getter = {
  | index |
  Plazy {
   if (~n.isNil) {
    Pseq([Rest(1)]);
   } {
    ~n[index] ?? (~n[0] ?? Pseq([Rest(1)]));
   };
  };
 };

 n0 = note_getter.(0);
 n1 = note_getter.(1);
 n2 = note_getter.(2);

 right = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n1, n2] ++ (([ n0, n2, n1, n2] + 12)!2).flatten),
  \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
 );
 left = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12),
  \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
 );
 if (~player.notNil) { ~player.stop; };
 ~player = Pn(Ppar([right,left])).play;
});
)
)

Let me know if have ideas to enhance the system!

Walsh-Hadamart transformations in supercollider

Problem

Practically the whole world is using Fourier Transforms to decompose sounds into sums of sine waves. The Fourier transform then can be edited, and transformed back to the time domain to hear the effects of the editing. One question that naturally arises is if perhaps ways exist to decompose sounds as sums of something other than sine waves.

Approach

Well, as it turns out there are infinitely many ways to decompose signals into sums of other signals, and one that personally intrigues me is the Walsh-Hadamard transform which decomposes signals into sums of pulses (square waves). Walsh functions were already known and used around 1890, but it took until 1923 for Walsh to formally examine them in a mathematical context. Most of the work on Walsh functions in the context of audio processing was done in the nineteen-seventies. The fact that it is no longer popular may indicate that the results were nothing spectacular, but that shouldn't stop us from experiencing it first-hand.

Walsh functions

With the help of mathematics it can be established that we need only a subset of all possible square waves to decompose and reconstruct any signal. These waves are now known as "Walsh functions" and they can be ordered by sequency. (Note: not frequency!). Sequency is a number that corresponds the number of zero-crossings the pulse makes in the time base. Here's a representation of some Walsh functions ordered by their sequency. Sequency is not expressed in cps (cycles per second, also known as Hz), but in zps (zero-crossings per second).

If the signal is only 2 samples long, 2 Walsh functions suffice to perfectly reconstruct any such signal. If the signal is 4 samples long, 4 Walsh functions suffice to perfectly reconstruct any such signal. In general, for a signal of 2^L long, you need to combine up to 2^L Walsh functions to perfectly reconstruct any such signal. This is similar to the discrete Fourier transform where the original signal and the transformed signal both have the same length.

Walsh functions always start with +1 as their first component. For completeness I should probably mention that the even sequencies are sometimes called CAL functions, whereas the odd sequencies are called SAL functions.

WAL(2n, t) = CAL(n, t)    n = 1, 2, ...
WAL(2n-l, t) = SAL(n, t)  n = 1, 2, ...

As with sines and cosines, CAL and SAL are in essence time-shifted versions of each other.

Here's an example of decomposing a 4 sample signal into a linear combination of Walsh functions:

The signal [-1, 1, 0, -2] can be decomposed using level 2 sequencies:

[-1,1,0,-2] = -0.5*[1,1,1,1] + 0.5*[1,1,-1,-1] -1*[1,-1,-1,1] + 0*[1,-1,1,-1]

or in words:

[-1,1,0,-2]  = -0.5*sequency0 + 0.5*sequency1 - 1*sequency2 + 0*sequency3

Note that this formula shows how to convert from Walsh spectrum back to time domain using the (known) sequencies.

Ok, so how did you find that combination? Can you always do this? Is there always only one possible combination?

I'll skip the mathematics, but yes, it can always be done, and there's always exactly one possible decomposition. Some very smart people invented an efficient way to find this decomposition, and the efficient way is known as the "Fast Walsh Transform". Explaining the transform in detail, however, is way out of bounds for this article. Please refer to some external reference for the juicy details.

Looking at the example above, the fast Walsh transform of [ -1, 1, 0, -2] should give [ -0.5, 0.5, -1, 0].

And we can go back from [ -0.5, 0.5, -1, 0] to [ -1, 1, 0, -2] by using the sequencies as we did in the example above.While we did the calculations by hand in the example above, the fast Walsh transform actually has a wonderful property: it is its own inverse (except for some constant factor).

This means that if you apply the Walsh transform to a signal, you get the Walsh spectrum, and if you then apply the Walsh transform to that Walsh spectrum again, you get the original signal again (apart from some constant factor). 

What does any of this have to do with audio or supercollider?

Well, in itself nothing really.  But we can propose some experiments with this transform.

Remember that what we are really doing here is decomposing signals (think: sounds) into weighted sums of Walsh functions (think: square waves). Square waves happen to be a basic waveform used in subtractive synthesis (think analog synths!) so now it starts to sound kind of interesting doesn't it?

We already have a way to decompose any sound into a sum of square waves, and to go back from these square waves to the original signal (time domain). What if, just before we go back to the time domain, we modify the Walsh spectrum first?

What is the effect on a sound of removing all the fast square waves (= high sequencies)? (For lack of a better word, you could call it a Walsh-Low-Pass-Filter). What is the effect on a sound of removing the slow square waves (= low sequencies)? (kind of Walsh-High-Pass-Filter). What is the effect on a sound of setting all Welsh spectrum values to 0 if they happen to be smaller than some threshold? (This could be the core algorithm of some lossy data compression scheme). What is the effect on a sound of shifting the Welsh spectrum values to the left/right (a kind of Walsh-Pitch-Shifting). Can we synthesize interesting sounds by making up a new Walsh spectrum (kind of additive synthesis with square waves)? What does a walsh filter sweep sound like? What do you get if you reinterpret the Walsh spectrum as a Fourier spectrum or vice versa? Can useful/beautiful visualizations be derived from the Walsh spectrum?

It is to be expected that the auditory results will be wildly different from what we are used to hearing in transformations based on the Fourier transform (classical high-pass and low-pass filters e.g.), but that should be all the more reason to try it out, shouldn't it? Maybe you can think of really cool new applications made possible by using the Walsh transform in audio context? If so, be sure to comment :)

Walsh transform in supercollider

Before we can play with audio, we need a way to calculate the fast Walsh transform in supercollider. Since I don't know how to write UGEN's yet, I will do some calculations in the language for now.

Here's a pretty straightforward translation of this c implementation:
This code is also available on https://sccode.org/1-5bD.

If we evaluate the following code in scide:

~walsh_transform.(values:[-1,1,0,2]);

we get back the expected result:

[0.5,-0.5,0,-1]

And to check that the inverse transform works as expected:

~walsh_transform.(values:~walsh_transform.(values:[-1,1,0,2]), rescale:false);

gives the original signal:

[-1,1,0,2].

Bring on the sounds!

This walsh, this walsh, this walsh, this walsh...

My first interest is in hearing the timbres of the Walsh functions. by themselves. So let's listen to some of those. I'll take the 256 Walsh functions from level 8 (calculate them by applying the inverse walsh transform on a spectrum containing a single "1"), and concatenate 100 copies of each into a (stereo) buffer and then play the buffer.

The resulting tones pretty much sound like pulsewidth modulated pulse waves because, obviously, they *are* pulse waves.

This article has been more than long enough for now. If there's any interest in the subject I may prepare a follow-up article in which we experiment with Walsh-transform based filters.

Using the midi tuning specification (MTS) standard in supercollider

Problem?

I have a hardware synthesizer Dave Smith Instruments Rev2 ( (aka DSI Rev2)with support for MTS (=midi tuning specification) and I'm intrigued by its possibilities. How can I reprogram my synth's frequencies from supercollider? In a next post I may or may not (depending on how fast I can solve some annoying bugs :) ) explain how to download scala tunings and keyboard mappings with the Rev2 from supercollider, but for now let's concentrate on the low-level MTS messages. Most of the information in the article explain calculations, so it should also be usable for anyone trying to use MTS from any programming language.

Approach?

It will be explained how to assign random frequencies to all midi notes by composing a valid sysex message containing bulk tuning information. The explanation is somewhat math heavy, but every step is completely detailed which should help you implement something in your favorite programming language.

MTS?

Midi tuning specification is a protocol that allows users of synthesizers to assign any frequency to any midi note. You can reverse your keyboard if you like, or you can assign random frequencies to all keys, or you can explore the wonderful world of microtonality and xenharmonics. The way to assign any frequency to any midi note is by sending midi sysex messages.

Sysex?

Sysex is a hook in the midi protocol that allows for sending anything over midi. "Anything" is to be interpreted as follows: "anything that fullfills the demands made on bytes sent over MIDI, i.e. the highest bit is always 0". This constraint means that in some cases data has to be encoded in a form that is suitable to be sent over midi, and this is also the case for MTS messages. We'll go over the calculations in excruciating detail in the rest of the article.

With some exceptions, sysex messages are not standardized. Any manufacturer can implement its own sysex messages. DSI, e.g. uses sysex to perform OS updates on their instruments. Such sysex messages typically are not publicly documented.

The sysex messages used for sending tuning information, on the other  hand, are standardized. They come in a few flavors. There's a message to request an synth to send over its tuning to a computer, but the DSI instruments appear not to respond to those messages, so I guess they are not supported.  There's another message to send tuning information from the computer to a synth, and that works well. Still other messages exist to retune a few notes in real-time, but I haven't tried those yet.

MTS message structure

MTS message structure is not very complicated, but it takes a bit of explanation. The full explanation can be read in the midi tuning standard spec, but it might contain some mistakes (currently under investigation).

First of all let's explain the general message layout. 

F0 F7 7E 7F 08 01 IDX C1 C2 .. C16 TA1_0 TA2_0 TA3_0 .. TA1_127 TA2_127 TA3_127 CS F7

• F0 F7 appears at the start of any sysex message
• 7E specifies that it's a non-real-time message.
• 08 specifies that we're about to send tuning information
• 01 specifies that we're about to send a bulk request, i.e. tuning information for all midi notes
• IDX specifies a 0-based tuning index, this corresponds to the tuning index selected in DSI's global parameters menu, alternative tunings menu entry
• C1 .. C16 are 7-bit ascii codes for the characters in the name of the custom tuning
• TA1_0 TA2_0 TA3_0...TA1_127 TA2_127 TA3_127 are the actual tuning information. Tuning information for each midi note is described using three bytes, to be explained below.
• CS is a checksum, also explained below
• F7 signals the end of the sysex message

TA bytes tuning information structure

How do the three bytes TA1, TA2, TA3 encode tuning information for a midi note? Well, the first of the three bytes, TA1 is a number that says which lower midi note number is closest to the desired frequency, if the instrument would be tuned in 12-Tone equal temperament (i.e. freq  =  2^{(midinote-69)/12}*440 Hz or vice versa midinote  =  12*log₂(freq/440 Hz)) + 69, where midinote is a number 0-127, and freq is a float.

The bytes TA2 and TA3 then encode the difference in "cents" between the frequency indicated by the midi note number in TA1 and the desired frequency. The difference in cents between two frequencies f1 and f2 can be calculated as n_c  =  1200*log₂(f2/f1)), but supercollider provides some built-in functions to hide these formulas.

In the MTS message, 128 such triples TA1,TA2,TA3 are sent, one for each midi note. 

So if you wanted to retune the 50th midi note to the value of 670 Hz, you'd have to put as 50th triplet in the message the following bytes:

• the closest, lower midi number corresponding to 670 would be 76 (midi note 76, on a 12-TET tuned instrument, corresponds to a frequency of  2^(76-69)/12*440 = 659.26 Hz ). The way to calculate this 76 is by taking the floor of the formula that transforms frequency to midi note number: floor(12*log2(670/440 Hz) + 69) = 76. In hexadecimal, 76 is 4C. 

TA1_50 = 76 (or hex: 0x4C)

• the difference in cents between 659.26Hz and 670Hz would then be 1200*log₂(670/659.26) = 27.98908618457 cents. This difference in cents will always be < 100, because the reference frequency is the closest lower midi note (and the difference between two successive midi notes in 12 TET, is always exactly 100 cents).
• We have 2 bytes TA2 and TA3 to encode this number 27.98908618457. Because in midi bytes, the high bit is reserved to indicate commands instead of data, that gives us a total of 14 bits (= 0 to 2^14-1) to encode a number in the interval [0, 100] (I include 100 because the difference in cents might e.g. be 99.995). In other words, you get a resolution of 100 cents / (2^14) = 0.0061 cents. That is what we call super high precision :) 
• If you map number 27.98908618457 from the interval [0,100] to the interval [0, (2^14-1)], by means of the formula 27.98908618457 * (2^14-1)/100 ,you get 4585.4519896181, or after rounding 4585. This 4585 is the number that must be encoded in TA2 and TA3. This happens with bit masking operations:

TA2_50 = 4585 >> 7 = 35 (or hex: 0x23)

TA3_50 = 4585 & 127 = 105 (or hex: 0x69)

Checksum calculation

• The checksum is calculated as a XOR between all the bytes in the message, except for the sysex header (F0 F7), the checksum itself (obviously) and the end of sysex byte at the end (7F).
• Gotcha: at the end of the checksum calculation, be sure to "and" the result with 127 (=0x7F) to mask out the highest bit, otherwise you end up with an invalid MIDI message. This is not explicitly mentioned in the MTS standard, but it's implicitly assumed for all MIDI data bytes.

Gotchas

• Beware of limited precision of floating point operations. In some cases, it may lead to unexpected results and invalid sysex messages as a result. I encountered such problems in my supercollider implementation of the tuning message and put in a workaround.
• Be sure to check and double check that midi note numbers used as TA1 do not become negative or >127. Failing to do so will cause you hours of debugging (I should know by now!) to find out why sometimes - seemingly out of the blue - the synthesizer does unexpected and weird things when sending a new tuning (like randomly switching to a different program and changing some LFO settings) or why sending a new tuning doesn't seem to have any effect.
• The MIDI MTS spec states that sending a tuning value with TA1, TA2, TA3 = 0x7F 0x7F 0x7F should be interpreted by the synth as "do not change tuning", but it seems that DSI instruments interpret it as "tune to the highest possible frequency" instead. For this reason the code currently just puts the standard note where no note should be present.

Supercollider code

Note that the following screenshot is part of a supercollider class, part of a larger system to communicate with the DSI prophet rev2 hardware synthesizer (the code can be found here: https://github.com/shimpe/sc-prophet-rev2 ), so you'd have to change some minor syntax details to get it working as a regular function (click the image to enlarge, or - even better - copy paste it from github (which may have a more recent version): https://github.com/shimpe/sc-prophet-rev2/blob/master/Classes/ScalaCalculator.sc).

Panola: a supercollider PAttern NOtation LAnguage

Problem?

In supercollider, one of the best ways to schedule notes over time is by using the pattern system. The pattern system is very flexible but the flexibility can make it a bit hard to use. 

In a typical pattern specification, also known as a Pbind, every dimension to your sound event is independent from every other event. This is just a fancy way of saying that you need to specify volume separately from duration, separately from midinote, etc.

If you want to delete a note in the middle of your piece, you have to dig through all these keys in the pattern specification to find the right stuff to remove or add so as to make sure that all the rest still sounds as intended.

Solution?

By grouping all musical information together and extracting the required keys from that central specification, we can change information in one place and keep all the resulting keys in sync automatically.

Thinking about this, I thought back of a creation I made last year: the midi specification language (MISPEL, see https://github.com/shimpe/expremigen ). I dreamed about having this system available in supercollider. I didn't quite get there yet, but now at least a subset of Mispel is available in supercollider and that subset is known as Panola (pattern notation language), available from https://github.com/shimpe/panola 

Tutorial?

(For now this is a literal copy of the tutorial on http://sccode.org/1-5aq - feel free to go there, it has syntax coloring :) )

// Panola is a way to extract Pbind keys from a concise specification.
// This makes it easier to compose "traditional" music with Pbind, with a lot less
// headache trying to keep the different keys in sync
// It's the type of system I've missed since my day one with supercollider.

// First things first. To install Panola:

Quarks.install("https://github.com/shimpe/panola");

// Now you can get the help document by typing ctrl+D with the cursor on the word
// Panola in the next line

Panola.new("a4");

// Let's start with the "Hello world" of Panola: a simple scale.
// The numbers indicate octaves.
// You don't need to repeat octave numbers if they don't change between notes.
(
~ex = Panola.new("c4 d e f g a b c5");
~player = ~ex.asPbind.play;
)

// asPbind takes a synth name as parameter (which defaults to \default).
// So the above is equivalent to
(
~ex = Panola.new("c4 d e f g a b c5");
~player = ~ex.asPbind(\default).play;
)

// instead of calling a single "asPbind" you can also extract all information separately
// like this you have optimal flexibility in what you want to use from Panola
(
~ex = Panola.new("c4 d e f g a b c5");
~pat = Pbind(\instrument, \default, \midinote, ~ex.midinotePattern, \dur, ~ex.durationPattern, \amp, ~ex.volumePattern, \tempo, ~ex.tempoPattern, \lag, ~ex.lagPattern, \legato, ~ex.pdurPattern);
~player = ~pat.play;
)

// You can make chords using angular brackets. Only note properties of the first
// note in the chord (other than octave number and note modifier (see later)) are
// taken into account.
(
~ex = Panola.new("<c4 e> <e g> <c e g c5>");
~player = ~ex.asPbind.play;
)

// You can use modifiers on the notes:
// # for sharp, x for double sharp, - for moll, -- for double mol
(
~ex = Panola.new("c4 d- e f# gx a# b-- c5");
~player = ~ex.asPbind.play;
)


// With underscores you can indicate rhythm.
// The last used rhythm value is reused until a new one is specified:
// Here's four quarter notes (_4) followed by four eighth notes (_8).
(
~ex = Panola.new("c4_4 d e f g_8 a b c5");
~player = ~ex.asPbind.play;
)

// You can use one or more dots to extend the length of the rhythm, as in traditional notation.
(
~ex = Panola.new("c4_4. d_8 e_4 f g_16 a_4.. b_4 c5");
~player = ~ex.asPbind.play;
)

// You can also use multipliers and/or dividers to change the length.
// E.g. here we use it to create a note that lasts for three eighths
// (c4_8*3) and to create tuplets (e_8*2/3 f g). Remember that last
// duration/rhythm indication is reused until a new one is specified.
(
~ex = Panola.new("c4_8*3 d_8 e_8*2/3 f g f_16 e f e g_4 b_4 c5");
~player = ~ex.asPbind.play;
)

// Now we come to the animated property system. We can attach properties to the notes and animate them over time.
// For now two types of animation are supported: linear interpolation and fixed value.
// To indicate linear interpolation, use curly brackets {}. E.g. here we let the tempo gradually increase from 80 bpm to 160 bpm:
(
~ex = Panola.new("c4\\tempo{80} d e f g a b c5\\tempo{160}");
~player = ~ex.asPbind.play;
)

// Different properties can be combined. Here we let the volume go up until the middle of the phrase, then let it go down again,
// while tempo is rising from 80 bpm to 160 bpm.

(
~ex = Panola.new("c4\\tempo{80}\\vol{0.2} d e f g\\vol{0.9} a b c5\\tempo{160}\\vol{0.2}");
~player = ~ex.asPbind.play;
)

// If you want to use the fixed values, use square brackets instead. You can switch between fixed and animated everytime
// you specify a new property value. In the next example, tempo remains at 80 bpm until we come to note a. At that point,
// it jumps to value 100 bpm and gradually increases to 200.
(
~ex = Panola.new("c4\\tempo[80] d e f g a\\tempo{100} b c5 d e f g a b c6\\tempo{200}");
~player = ~ex.asPbind.play;
)

// Using pdur (think: played duration), we can indicate the difference between staccato and legato.
// Here we slowly evolve from very staccato to very legato:
(
~ex = Panola.new("c4_8\\pdur{0.1} d e f g a b c5 d e f g a b c6\\pdur{1}");
~player = ~ex.asPbind.play;
)

// Using lag we can modulate lag. This can be a way of creating a rubato feeling.
// Linear interpolation is not ideal for this purpose, but it's better than nothing at the moment.

(
~ex = Panola.new("a5_8\\tempo[120]\\lag{0} b c6 a5 e d c5 d e c a4 g#4\\lag{0.5} "
"a4_8 b c5 a4 e d c4 d e c a3 g#3 a b c4 d e g# a_2\\lag{0}");
~player = ~ex.asPbind.play;
)

// In addition to using predefined properties like tempo and lag, you can also use user
// defined properties, e.g. here we animate a property called "myprop".
(
~phrase = Panola.new("c d\\myprop{0.1} e f g a\\myprop{0.6}");
~pattern = ~phrase.customPropertyPattern("myprop"); // extract only myprop values as a pattern
~stream = ~pattern.asStream;
10.do({
| i |
~stream.next.postln;
});
)
// make a pbind in which the myprop appears as one of the keys, with a default value of 0 for myprop
(
~pbind = ~phrase.asPbind(\default);
~stream = ~pbind.patternpairs[13].asStream;
10.do({
| i |
~stream.next.postln;
});
)
// make a pbind in which the myprop appears as one of the keys, with a customized default value of 0.4 for myprop
// (such default values are used if no values for myprop are specified yet, e.g. in the beginning of a Panola string,
//  before any myprop is defined).
(
~pbind = ~phrase.asPbind(\default, custom_property_defaults:Dictionary.newFrom(["myprop", 0.4]));
~stream = ~pbind.patternpairs[13].asStream;
10.do({
| i |
~stream.next.postln;
});
)
// make pbind in which only the standard panola keys are included
(
~pbind = ~phrase.asPbind(\default, include_custom_properties:false);
~pbind.patternpairs.postln;
)

// These custom properties can be e.g. used to drive synth arguments
// The 303 synth used below is reused from https://sccode.org/1-4Wy
// which in turn is based on code from Lance J. Putnam
(
s.waitForBoot({
var line;

SynthDef (\sc303 , {  arg  out=0, freq=440, wave=0, ctf=100, res=0.2,
sus=0, dec=1.0, env=1000, gate=1, vol=0.1;
var  filEnv, volEnv, waves;
volEnv =  EnvGen .ar( Env .new([10e-10, 1, 1, 10e-10], [0.01, sus, dec],  'exp' ), gate, doneAction:2);
filEnv =  EnvGen .ar( Env .new([10e-10, 1, 10e-10], [0.01, dec],  'exp' ), gate);
waves = [ Saw .ar(freq, volEnv),  Pulse .ar(freq, 0.5, volEnv)];
Out .ar(out,  RLPF .ar(  Select .ar(wave, waves), ctf + (filEnv * env), res).dup * vol);
}).add;

s.sync;

line = Panola.new(
"a2_16\\wave[0]\\vol{0.05}\\tempo{120}\\res{0.2}\\sus{0}\\env{1000}\\ctf{100} a a a1 a2 a a3 a2 a a a1 a2 a3 a2 b- g\\res{0.05}"
"a2_16\\wave[0] a a a1 a2 a a3\\sus{0.2} a2 a\\ctf{3000} a a1 a2 a3 a2 b- g\\res{0.2}"
"a2_16\\wave[0] a a a1 a2 a a3 a2 a a a1 a2 a3 a2 b- g\\res{0.01}\\sus{0}\\env{10000}\\ctf{10}"
);
~player = line.asPbind(\sc303).play;
});
)

Baking sound in supercollider

Problem?

I want to synthesize gorgeous pad sounds in supercollider. This is possible using additive and/or subtractive synthesis, but it takes a lot of CPU power.

Approach

't Is the season to be bakin' fa-la-la-la-laaaaa la-la-la-laaaaa. How about we pre-render the sound into a wavetable and then loop the generated wave table? This moves most of the work to startup time. In the blender 3d program, pre-rendering heavy calculations is known as "baking".

But... won't we lose real-tome creative possibilities then? After all, you cannot modify the partials in a prerendered wavetable anymore?

Well... yes, you will lose some creative possibilities, but we can easily reintroduce some by adding amplitude envelopes, filters and filter envelopes and effects like reverb or phasing and flanging.

PadSynth

One of the more exciting pad generation algorithms no doubt is Paul Nasca Octavian's PadSynth algorithm. It is explained in detail here. I will use it to demonstrate the approach.

In essence it takes as input a basic sound, specified as a set of partials with associated relative volumes. 

It then enriches that spectrum by replacing each of the partials with a fuzzy region containing smeared out partials. This is the equivalent to adding many slightly detuned copies of the original signal, which is how sounds can start to sound a lot warmer. Since human hearing is less sensitive to small frequency differences for high frequencies than for low frequencies, the spreading of partials is made wider in the higher frequency bands than in the lower frequency bands. PadSynth also sets the phases of each of the detuned copies of the signal to random values which helps in reducing undesired comb filtering effects.

PadSynth in supercollider

Supercollider does not come with an implementation of PadSynth, but it is not too difficult to convert the pseudo-code from the detailed explanation into working code. A post by Donald Craig on the supercollider mailing list provides an excellent starting point.

I modified the algorithm a bit as follows: I calculate two different wavetables per octave over 12 octaves (this is probably a bit overkill - feel free to change it). This is an intensive calculation and takes quite a lot of memory. You can certainly reduce this calculation time and required space by reducing how many octaves you want to use, by reducing the number buffers per octave and by reducing the size of the pregenerated wave table (I generate wavetables of around 5 seconds long each).

While playing, the pitch can be chosen by looping over the generated wave tables with a different speed. By varying the speed on-the-fly one can also implement pitch bending or glissando's. All of this is illustrated in the sample code below, in which I will be using supercollider's excellent pattern system to drive a cello-like PadSynth pad. The code can also be found on sccode.org.

Driving the synth from supercollider patterns

Because the synth loops over one of many pre-generated buffers, we'll have to inform the synth about which buffer to use for a given note. For this reason you'll see some calculations being done in the Pbind and Pmono in the sample code. If you decide to generate less buffers per octave, you will also have to adjust these calculations.

More in detail: the parameter \mynote contains a midi note number. This is the note we will want to hear. From that parameter, another parameter \myreference is calculated. This contains another midi note, namely the closest midi note number that was used as a reference to pre-render a wave table: I generate 2 wavetables per octave and each of these wavetables have a reference midi note number. This information is needed in the synth to determine how fast to loop over the buffer. \referencefreq contains the same information but expressed as a frequency in Hz instead of as a midi note number. Remember that I generated two wavetables per octave, for 8 octaves. The parameter \buffer contains the buffer number to use. The buffer being used is the one that was generated with a reference midi note closest to the desired midi note. This information is needed in the synth to know which buffer to loop over. 

The full code is a bit long to paste into this article, so please head over to sccode.org to get it!

If sccode.org is down, there's still github as a backup :) 

Audio transitions in supercollider

Problem?

In the domain of images and videos, programs like powerpoint and video editors provide a wide range of "transitions". If you think of powerpoint, e.g., you can switch from one slide to the next directly ("cut"), or you can gradually fade out the first one and fade-in the second one, or you can push in, push out, wipe according to a shape, split, reveal, random bars, shape, uncover, cover, flash, morph, ... too many to name.

In the audio domain, however, the options seem quite limited. I'm only aware of direct cuts and cross-fading as audio effects that are offered by default. Can we think of other audio transitions as well?

Approach

In this blog post I will propose a few other audio transitions, as a proof of concept. The code can be found on http://sccode.org/1-57H.

Setting up supercollider

To illustrate the transitions, let's first create two sounds that we want to transition between.

\sound1 is a sine wave with randomly varying frequency (frequency changes 4x per second).

\sound2 is a sawtooth with varying duty cycle. Both sounds are different enough that we can hear the transitions taking place.

I let the 2 sound producing synths output their audio to audio busses.

Next to the sound producing synths I have a bunch of transition synths that calculate the transition between \sound1 and \sound2 as a kind of audio effect. Each transition synth takes a pos argument. Setting it to -1 corresponds to only listening to \sound1, whereas setting it +1 corresponds to only listening to \sound2. For values between -1 and 1 the transition is taking place.

For demo purposes, the pos argument in the transition synths is ignored and instead driven from a fancy counter made specified by an envelope (Env). This fancy counter ensures that supercollider first plays \sound1 for 5 seconds, then gradually transitions to \sound2  during 5 seconds, then keeps playing \sound2 for 5 seconds, then transitions back to \sound1 during 5 seconds and finally plays \sound1 again for 5 seconds.

Groups are needed to ensure that the transition synths are executed on the server after the sound producing synths. Otherwise we wouldn't get audio output. (After all, it's impossible to fiddle with a sound that isn't calculated yet.). 

In the fork section we will instantiate all effect synths one by one so we can hear all of them in succession. In the first screenshot only the "direct cut" is added. This is a transition that basically does nothing: it just stops the first sound and starts the second one in the place.

Familiar effects

cut

A cut (in audio this is sometimes called a "butt splice") is where you stop one sound and start another sound. This basically means: "no effect". In supercollider this can be implemented by stopping one sound and starting another one. This is already illustrated in the screenshot above.

crossfade or xfade

A crossfade happens when you graduallly lower the volume of one signal while gradually increasing the volume of a second signal. For a while both signals sound simultaneously. This is supported in virtually every digital audio workstation (DAW) in existence. Often one can choose whether to change the volume keeping a "constant gain" (linear curve), "constant power" (square root curve?) or even other types like "exponential" (using logarithmic curves).

In the following screenshot I indicated what was changed to add a new effect. In the next effects I will only show the new effects synths. You can easily update the fork section yourself.

So for cross fading with equal power you could do something more like

Seldom explored effects

Do we have to stop at direct cut and crossfade? Of course not... let's try something else. Note: these techniques may actually be well established, it's just that I've never come across them "in the wild". But then, also not many people use transitions in powerpoint and video editing because if you overdo them, they tend to lower the overall end-user experience.

HPF curtain

In this transition, we increase the filter frequency of a high pass filter (HPF) for the signal that is to disappear, while we decrease the filter frequency of a high pass filter for the signal that is to appear. The implementation of this transition is not perfect as it has some audible artifacts on positions close to -1 and +1. It might help to use a high pass filter with steeper roll-off, or to combine this technique with some amplitude control.

LPF curtain

Very similar to the previous transition, we decrease the cut-off frequency of a low pass filter (LPF) on the signal that must disappear, and crank up the cut-off frequency of a low pass filter on the signal that must appear instead.

Wash out

The sound that is to disappear drowns in reverb whereas he other sound emerges from the reverb.

Pixelate

 The sounds become coarser and then more crisp again. This fragment features some complication involving Select.ar because LFPulse does not behave mathematically correct for duty cycle width=0 and width=1 (it generates audible spikes, which ruin its function as an audio stream selector), so near the duty cycle = 1 and 0, I had to select pure versions of the audio.

Push out

The new sound shifts in from the left side and pushes out the old sound to the right.

Can you think of other transitions?

Scoring a movie with supercollider, blender, python and osc - part II: keyframing and animation curves

Problem?

In the previous part of this tutorial, I described a way to insert markers on blender's timeline, and to use blender's python scripting abilities to convert these markers into OSC messages which then can be interpreted in supercollider to perform commands (e.g. start and stop patterns or sound effects). This approach works very well for sending discrete commands: things like start this, stop that. If you haven't read that part yet, please go read it as you will need to understand the concepts explained there to follow part II.

In this blog post, I want to show how one can use blender's powerful keyframing system with animation curves to continuously update parameters in supercollider sketches. This technique adds extra possibilities for driving supercollider from blender.

Approach?

In the blender scene, I will add custom properties, and animate those via keyframes and animation curves. I will extend the python script I developed in part I to send the parameter value updates to supercollider.

What is a keyframe?

A keyframe represents the value (the state) of one or more parameters at a moment in time. Suppose you have an X-position of an object in a 3d scene. At time frame=0 you can set X=0 and insert a keyframe on frame=0. At time frame=100, you can set X=10 and insert a new keyframe on frame=100. Now for frames between 0 and 100, the value of x can smoothly vary between 0 and 10, that is, the values of x can be interpolated between the keyframes (you can also ask the system for an abrupt change instead of interpolation). The interpolation can be linear, or can include "easing".

Adding custom properties

Blender is extremely keyframeable. Literally every parameter you see in the UI can be keyframed. Blender is also very extensible: users can add new parameters ("custom properties") in the UI. And those new parameters can be keyframed. 

To add a custom property, first switch to blender's default view

Then, on the right hand side, go to the scene tab, and in the custom properties section, click the Add button.

As we did in part I of the tutorial, we will again use a naming convention for the properties so that our python script knows what to send to supercollider from the property name. The proposed naming convention is that a property named SC_something42 wil be sent to supercollider as osc command /something42 with as argument the property value. We will make the python script so that only changes in values are sent. Values that don't change from one frame to the next will not be resent to supercollider. The intention is to avoid creating an avalanche of useless OSC messages.

After clicking the add button, we can set up our property using the naming convention. Suppose I want to automate a frequency in some synth. A possible custom propertyname could be SC_animfreq. This would trigger sending an OSC message /animfreq to supercollider with the value of animfreq as argument. I've set the default value to 440, and the min and max to 20 and 20000 respectively.

Now on the blender timeline, navigate the playhead to - say - frame number 10 (the easiest way is to type 10 in the current frame edit box), set the value of customproperty animfreq to - say - 220. Hover your mouse cursor over the value 220 of animfreq and press the letter "i" (for "i"nsert keyframe). The edit box with value 220 should change color to indicate that a keyframe has been inserted for the custom property.

Next, move the playhead to - say - frame 100, change the value of the custom property to 880, hover the mouse cursor over the 880 and "i"nsert another keyframe.

Now as you drag the playhead between frames 10 and 100, watch the custom property value box. You should see the number change gradually. This is the linear interpolation that happens between keyframes by default.

If we want to alter this behavior of linear interpolation, we need to switch to the "graph editor". A full explanation of everything that an be done in the graph editor falls outside the scope of this blog post (i.e. you'll need to find some blender tutorials about the graph editor on youtube if you want to dive deeper - let's just say that you can do some funky stuff).

After some zooming in x and y direction you can see the linear interpolation drawn as a curve.

By pressing "T" while hovering the mouse cursor over the graph area you can change the interpolation type. Since we're animating a frequency, it makes sense to set the interpolation type to exponential. Be sure to look for tutorials on F-curve modifiers to see some fancy stuff that is possible with the curves. For the purpose of this tutorial we won't go any deeper.

So far we have added an animated custom property. Now let's tell blender to send it to supercollider in the form of OSC messages. To do so I will extend the python script developed in part I of the tutorial. The additions with respect to part I are marked in the figure.

The OSC messages can be received in supercollider in exactly the same way as in part I. The received values then e.g. can be used to set the freq argument in a synth (or the density in some cloud of sound grains, or whatever you feel like doing).

A blender file and supercollider file with a simple demonstration can be downloaded from google drive.