Driving the GODOT game engine with OSC messages

Problem?

Note: I've tested this code with a very recent version of GODOT compiled from source code. Your mileage may vary.

In interactive art installations, an often used protocol to communicate between computer processes is the open sound control (OSC) protocol. Despite the name "sound" in OSC, the protocol is actually completely general purpose. In my opinion a game engine like the godot engine would benefit tremendously from built-in support for the OSC protocol because it opens up the engine to creative uses outside the gaming domain. On of my own use cases is to visualize in real-time musical events that are algorithmically generated (also in real-time) in supercollider.

The GODOT development team, despite several requests from users, chose not to support OSC directly and provide a high-level networking interface which they feel is more appropriate for their game engine (but which only works with their game engine). 

As a result, some users have attempted to write add-on modules using the GdNative C++ add-on mechanism to extend the GODOT engine. Such modules often have the drawback of limited portability and requiring users to set up a C++ building environment. In addition, the API that needs to be used by the add-ons could be changing drastically between GODOT versions. 

Therefore, I was hoping to find a more portable solution. Luckily GODOT also provides some lower level networking classes, which allow third-party developers to add some support for unsupported protocols.

OSC is typically a thin layer on top of UDP networking (sometimes TCP is used, but most often UDP is chosen for reasons of performance), and so it should be possible to add a simple OSC implementation in GDScript directly.

Note: to make sense of this blog post, you need to have a little bit of working knowledge about the GODOT game engine, at the minimum, you need to know what nodes are.

Will it perform super fast? 

Probably not. But often you don't really need to send tons of data over the network to drive your visualization. Think about optimizing your data streams.

Will it be easy to use?

I think it is extremely easy to use. But feel free to try it out and judge for yourself.

How can I use it?

Glad you ask. Here's the big picture:

To the root note of our GODOT sketch (usually a Node2D for 2d games, or a Spatial for 3d games), I attach a script that monitors the network for incoming OSC messages. The OSC messages are decoded and automatically dispatched to child nodes who requested to be kept informed about certain incoming messages.

Consider an example of a Node2D rootnode (which I renamed to RootNode in the IDE) with a Sprite child node. In the script attached to the Sprite, I only have to express my desire to be kept up-to-date about incoming OSC messages, e.g. here's the complete script for a sprite that can change position by receiving a /set/xy OSC message.

extends Sprite

func _ready():
   get_tree().get_root().get_node("RootNode").register_dual_arg_callback("/set/xy", get_node("."), "set_xy")

func set_xy(x, y):
   self.position.x = x
   self.position.y = y

This means that when the Sprite is instantiated, it registers itself with the root node ("register_dual_arg_callback"). It tells the root note that it wants its function "set_xy" to be called whenever a "/set/xy" OSC message is received over the network with two arguments (x and y) which in this example then are used to set the sprite position. There's also a register_single_arg_callback for functions that expect only one value (and nothing prevents you from adding more variants).

This simple code suffices to ensure that from now on, this particular Sprite will automatically react to incoming OSC /set/xy messages by updating its position.

E.g. in an environment that supports sending OSC message like supercollider, I can run the following snippet to update the sprite position:

(

b = NetAddr.new("127.0.0.1", 4242); // create the NetAddr
b.sendMsg("/set/xy", 650, 200);

)

The godot sketch automatically reacts to the OSC message by updating the sprite xy position to 650, 200, exactly what we requested the system to do by registering a callback. Needless to say the OSC message could perfectly come from another computer, tablet or phone (e.g. from the TouchOsc app or from Open Stage Control).

The value 4242 is the port number on which the root node is listening for incoming OSC messages. In the proof of concept code it's hardcoded to 4242, but you are of course free to change it or to make it configurable.

Ahm, ok. Show me this magic code that goes into the root node then?

Note: the code in this section is obsolete, and replaced by the code in part 2 of this article: https://technogems.blogspot.com/2021/11/driving-godot-game-engine-with-osc.html

Well... it's not the prettiest code (it's a proof of concept - things like IP address and port are hardcoded, but should be simple to change). It works well enough for me at the moment (but I've only done some basic experiments so far). In case I run into trouble, I may need to revisit the details. If you run into trouble, please let me know what happened and how you solved it (or just explain the problem and we can discuss to see if we can find a solution).

Here's the code I put in the root node. For now it only supports parsing integer, float, string and blob OSC messages. In the future maybe also OSC bundles and some other data types could be added, but even with only this simple subset of OSC supported, the possibilities are already endless.

Note that the call to OS.set_low_processor_usage_mode may not be portable across all platforms, but it's probably not strictly needed to make the system work (but don't take my word for it, I'm not at all experienced with GODOT on platforms other than linux).

The code for registering callbacks and dispatching probably can be made a bit more general, and there could be support for pausing, resuming and stopping reacting to OSC notifications but for demonstration purposes what I have here should suffice.

extends Node2D

var IP_CLIENT
var PORT_CLIENT
var PORT_SERVER = 4242 # change me if you like!
var IP_SERVER = "127.0.0.1" # change me if you like!
var socketUDP = PacketPeerUDP.new()
var observers = Dictionary()

func register_single_arg_callback(oscaddress, node, functionname):
 observers[oscaddress] = [1, node, functionname]
 
func register_dual_arg_callback(oscaddress, node, functionname):
 observers[oscaddress] = [2, node, functionname]
 
func _ready():
 OS.set_low_processor_usage_mode(true)
 start_server()
 
func all_zeros(lst):
 if lst == []:
  return true
 for el in lst:
  if el != 0:
   return false
 return true

func _process(_delta):
 if socketUDP.get_available_packet_count() > 0:
  var array_bytes = socketUDP.get_packet()
  #var IP_CLIENT = socketUDP.get_packet_ip()
  #var PORT_CLIENT = socketUDP.get_packet_port()
  var stream = StreamPeerBuffer.new()
  stream.set_data_array(array_bytes)
  stream.set_big_endian(true)
  var address_finished = false
  var type_finished = false
  var address = ""
  var type = ""

                # parse osc address
  while not address_finished:
   for _i in range(4):
    var addrpart = stream.get_u8()
    if addrpart != 0:
     address += char(addrpart)
    if addrpart == 0:
     address_finished = true
    

                # parse osc type list 
  while not type_finished:
   for _i in range(4):
    var c = stream.get_u8()
    if c != 0 and char(c) != ",":
     type += char(c)
    if c == 0:
     type_finished = true

  # decode values from the stream
  var values = []
  for type_id in type:
   if type_id == "i":
    var intval = stream.get_32()
    values.append(intval)
   elif type_id == "f":
    var floatval = stream.get_float()
    values.append(floatval)
   elif type_id == "s":
    var stringval = ""
    var string_finished = false
    while not string_finished:
     for _i in range(4):
      var ch = stream.get_u8()
      if ch != 0:
       stringval += char(ch)
      else:
       string_finished = true
    values.append(stringval)
   elif type_id == "b":
    var data = []
    var count = stream.get_u32()
    var idx = 0
    var blob_finished = false
    while not blob_finished:
     for _i in range(4):
      var ch = stream.get_u8()
      if idx < count:
       data.append(ch)
      idx += 1
      if idx >= count:
       blob_finished = true
    values.append(data)
   else:
    printt("type " + type_id +" not yet supported")

  if observers.has(address):
   var observer = observers[address]
   var number_args = observer[0]
   var nodepath = observer[1]
   var funcname = observer[2]
   if number_args == 1:
    nodepath.call(funcname, values[0])
   elif number_args == 2:
    nodepath.call(funcname, values[0], values[1])
    
func start_server():
 if (socketUDP.listen(PORT_SERVER) != OK):
  printt("Error listening on port: " + str(PORT_SERVER))
 else:
  printt("Listening on port: " + str(PORT_SERVER))

func _exit_tree():
 socketUDP.close()

Compared to the misery of adding, compiling, maintaining, porting, ... a GdNative module, I think this is pretty acceptable (at least for my use cases).

Making an arpeggiator in supercollider with patterns

Problem

Given some notes as input, generate a pattern making use of those notes. When the input changes, the generated pattern should also change. Most keyboards and synthesizers provide simple arpeggiators, but we'll be using supercollider which allows for generating the most complex patterns imaginable including generation of chords, polyphony, introducing random variations, etc
Code for the final piece of code in this post can be found on sccode.org: http://sccode.org/1-5cr

Approach

How can we convert a list of notes to an interesting arpeggio? Different possibilities exist, but we'll be using one of the more powerful abstractions available to supercollider users: the pattern system.

Patterns act as a kind of template for generating events, and events can be played to create sounds. This sounds exactly like what the doctor prescribed.

A simple example to get started

First let's assign some midi notes to a variable ~n:

~n = [60, 64, 67]; // c major chord

If you, like me, prefer to reason in note names instead, you can install the Panola quark:

Quarks.install("https://github.com/shimpe/panola");

Then you can write the following instead (it's a bit longer but, hey,  at least with readable note names):

~n = Panola("c4 e4 g4").midinotePattern.asStream.all;

Our task is to define a pattern that uses the notes in ~n and builds from them a simple arpeggio. So, given the input notes [c4, e4, g4] instead of playing simply the notes [c4, e4, g4] we'll generate a more interesting arpeggio [c4, g4, e4, g4]. When the input notes change to e.g. [c4, f4, a4] the arpeggio that is generated should change to [c4, a4, f4, a4].

(
s.waitForBoot({
    var arp = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([n0, n2, n1, n2])
        },
        \dur, Pseq([1,1,1,1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(arp).play;
});
)

This code requires some explanation:

• s.waitForBoot is a construct I use in pretty much every supercollider sketch I make. It will start the supercollider sound server if it wasn't started yet.
• Once the server is booted, the function that is passed to waitForBoot is executed. This function defines a pattern (also known as Pbind) "arp" and plays it.
• The full power of Pbind is available (meaning that you could e.g. generate midi events and send them to a hardware instrument), but for demo purposes we just instantiate the supercollider default instrument. This should generate some sound even if you don't own any fancy hardware (after all, supercollider is also a sound synthesis language and therefore very capable of generating its own sounds). Instantiating the default instrument is accomplished by specifying the key-value pair \instrument, \default in the Pbind.

Plazy is a filter pattern that allows to calculate a new supercollider pattern using a function.

In addition to selecting an instrument, we also need to generate the notes to be played. Remember that ~n is the input note list. First we try to extract the first 3 notes from the input note list. These are the notes that we will use to rearrange into our arpeggio.

The line ~n = ~n ?? [Rest(1)] checks if variable ~n is defined already (actually it checks if ~n is nil). If it is not defined, it is assigned a list of input notes consisting of a single Rest. Then I introduce some variables n0, n1, n2 to denote the first, second and third note in the input list respectively. It may happen that the input list contains fewer than 3 notes (e.g. if you initialize the ~n variable from midi input from a hardware device, someone might play a 2-note chord instead of a 3-note chord). In that case we don't want our code to crash. If the first note is missing, I replace it with a Rest. If the second or third notes are missing, they are replaced with the first note.

The function passed to Plazy returns a Pseq that generates our arpeggio consisting of the first, third, second and third input note: n0, n2, n1, n2. Pseq is a pattern that generates successive notes. By default the complete list of notes will be repeated once and then the pattern stops.

To generate an arpeggio we are not limited to only generating notes. We can also generate durations, volumes, legato-staccato, and a bunch of other properties. All these can be derived from the input note list, or they can be completely independent from them if so desired. In this first example, let's just give all notes equal duration and keep all other properties to their default values. Specifying durations in a pattern is done by using the \dur key.

I want the complete arpeggio to be finished in 2 beats, so I specify 4 (because there are 4 notes) equal relative durations of 1 and then normalize the values to make their sum == 2.

Finally, we need to stop any previous instances of the pattern that may be playing, and make our pattern start. This happens in the lines

if (~player.notNil) { ~player.stop; }; // call stop if not stopped already
~player = Pn(arp).play;

Note the use of the Pn pattern, to make our arpeggiator repeat indefinitely.

As soon as you redefine the ~n variable to a new value, the pattern (but only after the previous instance was completely finished) will use the new values in ~n and generate a new arpeggio built from the new notes, so e.g. while the pattern is playing, try evaluating the following lines one by one, listening how it changes the arpeggio.

~n = [60, 64, 67 ];

~n =[ 60, 65, 69 ];

~n =[ 59, 65, 67 ]; 

Exercise: adapt the code to make an arpeggio based on the first four notes of a list of input notes.

Exercise: adapt the code to use different durations for different notes

Fancier patterns

The simple arpeggio we generated above is already more complex than what many synthesizers and keyboards can do, but supercollider being supercollider this is just the tip of the iceberg available to us.

We can generate multiple patterns from a single list of input notes and play them together with Ppar. In addition, not all durations and amplitudes need to be the same. You could generate a complete 16 track auto-accompaniment from a simple list of input notes using this technique. Here's an example of a melody pattern with a bass line generated from the input notes. Note that I do some arithmetic on the notes (+12) to add an octave. In general you are not limited to using only the input notes given by the user. You can add any other note you desire, which can (but needn't) be derived from one of the notes in the input list.

(
s.waitForBoot({
    var right, left;
    ~n = ~n ?? [Rest(1)];
    right = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n1, n2 ] ++  (([ n0, n2, n1, n2 ] + 12)!2).flatten)
        },
        \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
    );
    left = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12)
        },
        \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(Ppar([right,left])).play;
});
)

Exercise: define some percussion instruments and add some percussion to the fragment.

Listening to midi input

Until now we've defined the ~n variable manually. But we can just well listen to a midi device and react to the incoming notes (or control change messages).
Before we can use midi devices, we need to initialize supercollider's midi system. To do so, evaluate the following code:

(
MIDIdef.freeAll;
MIDIClient.init;
MIDIIn.connectAll;
)

MIDIdef.freeAll will remove any midi handlers that may still be running. MIDIClient.init will initialize midi communication in supercollider and MIDIIn.connectAll ensures that we react to incoming midi msgs of all midi devices connected to the system.

Now we can install a midi handler that reacts to note on and note off messages. We will maintain an array of notes, and for each note in the array whether it's on or off. This array forms the basis from which we derive our list of input notes.

Note that in the following, the arpeggio keeps playing until we press another chord. If you want the arpeggio to stop when you release the midi keys, you can add

~n = ~note_table.selectIndices({|item, i| item != 0});
if (~n == []) { ~n = nil; };

in the note off handler.

(
MIDIdef.freeAll;
MIDIClient.init;
MIDIIn.connectAll;
)

(
s.waitForBoot({
    var right, left;

    ~note_table = 0!127;
    
    MIDIdef.noteOn(
        \mynoteonhandler, // just a name for this handler
        {
            |val, num, chan, src|
            num.debug("num");
            ~note_table[num] = 1; // update note table and update ~n
            ~n = ~note_table.selectIndices({|item, i| item != 0}).postln;
        }
    );

    MIDIdef.noteOff(
        \mynoteoffhandler, // just a name for this handler
        {
            |val, num, chan, src|
            num.debug("num");
            ~note_table[num] = 0; // update note table and update ~n
        }
    );

    right = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pxrand([ n0, n2, n1, n2 ] ++  (([ n0, n2, n1, n2 ] + 12)!2).flatten)
        },
        \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
    );
    left = Pbind(
        \instrument, \default,
        \midinote, Plazy {
            var n0, n1, n2;
            ~n = ~n ?? [Rest(1)];
            n0 = ~n[0] ?? Rest(1);
            n1 = ~n[1] ?? ~n[0];
            n2 = ~n[2] ?? ~n[0];
            Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12)
        },
        \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
    );
    if (~player.notNil) { ~player.stop; };
    ~player = Pn(Ppar([right,left])).play;
});
)

Note that now always a complete "iteration" of the pattern has to be finished before changes in the input note list create a new arpeggio. You cannot switch the arpeggio to a different chord in the middle of the arpeggio. How can we change this behaviour?

Change chords in the middle of the arpeggio

What if we want changes in the input notes to have immediate effect? Can we adapt the code to make the system react faster to changes? Well... it's supercollider so of course we can. Let's see how it can be done.
If we want immediate reaction of the system to chord changes, one approach is to replace n0, n1, n2 with patterns that reevaluate a function every time they are called. This function then performs a lookup of a note in our ~n variable, which is updated as soon as new midi notes are received.
Also I moved the midi initialization code inside the system because I don't really like having to evaluate multiple code blocks successively.

(
s.waitForBoot({
 var right, left;
 var n0, n1, n2;

 MIDIdef.freeAll;
 if (~midi_initialized.isNil) {
  MIDIClient.init;
  MIDIIn.connectAll;
  ~midi_initialized = 1;
 };

 ~note_table = 0!127;
 ~n = nil;

 MIDIdef.noteOn(
  \mynoteonhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 1; // update note table and update ~n
   ~n = ~note_table.selectIndices({|item, i| item != 0});
  }
 );

 MIDIdef.noteOff(
  \mynoteoffhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 0; // update note table and update ~n
   /*
   // enable next two lines only if you want arpeggios to stop playing
   // when you release the midi keys
   ~n = ~note_table.selectIndices({|item, i| item != 0});
   if (~n == []) { ~n = nil; };
   */
  }
 );

 n0 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   ~n[0] ?? Pseq([Rest(1)]);
  };
 };

 n1 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   Pseq([~n[1] ?? ~n[0]]);
  };
 };

 n2 = Plazy {
  if (~n.isNil) {
   Pseq([Rest(1)]);
  } {
   Pseq([~n[2] ?? ~n[0]]);
  };
 };

 right = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n1, n2] ++ (([ n0, n2, n1, n2] + 12)!2).flatten),
  \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
 );
 left = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12),
  \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
 );
 if (~player.notNil) { ~player.stop; };
 ~player = Pn(Ppar([right,left])).play;
});
)

Final cleanup

As a final cleanup we can remove some code duplication

(
s.waitForBoot({
 var right, left;
 var n0, n1, n2;
 var note_getter;

 MIDIdef.freeAll;
 if (~midi_initilized.isNil) {
  MIDIClient.init;
  MIDIIn.connectAll;
  ~midi_initialized = 1;
 };

 ~note_table = 0!127;
 ~n = nil;

 MIDIdef.noteOn(
  \mynoteonhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 1; // update note table and update ~n
   ~n = ~note_table.selectIndices({|item, i| item != 0});
  }
 );

 MIDIdef.noteOff(
  \mynoteoffhandler, // just a name for this handler
  {
   |val, num, chan, src|
   ~note_table[num] = 0; // update note table and update ~n
   /*
   // only enable the following lines if you want the arpeggio to stop as soon as you release the keys
   ~n = ~note_table.selectIndices({|item, i| item != 0});
   if (~n == []) { ~n = nil; };
   */
  }
 );

 note_getter = {
  | index |
  Plazy {
   if (~n.isNil) {
    Pseq([Rest(1)]);
   } {
    ~n[index] ?? (~n[0] ?? Pseq([Rest(1)]));
   };
  };
 };

 n0 = note_getter.(0);
 n1 = note_getter.(1);
 n2 = note_getter.(2);

 right = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n1, n2] ++ (([ n0, n2, n1, n2] + 12)!2).flatten),
  \dur, Pseq([1, 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ].normalizeSum*2)
 );
 left = Pbind(
  \instrument, \default,
  \midinote, Pseq([ n0, n2, n0, n2, n0, n2, n0, n2] - 12),
  \dur, Pseq([1, 1, 1, 1, 1, 1, 1, 1].normalizeSum*2)
 );
 if (~player.notNil) { ~player.stop; };
 ~player = Pn(Ppar([right,left])).play;
});
)
)

Let me know if have ideas to enhance the system!