Sunday, July 22, 2018

Using the midi tuning specification (MTS) standard in supercollider

Problem?


I have a hardware synthesizer Dave Smith Instruments Rev2 ( (aka DSI Rev2)with support for MTS (=midi tuning specification) and I'm intrigued by its possibilities. How can I reprogram my synth's frequencies from supercollider? In a next post I may or may not (depending on how fast I can solve some annoying bugs :) ) explain how to download scala tunings and keyboard mappings with the Rev2 from supercollider, but for now let's concentrate on the low-level MTS messages. Most of the information in the article explain calculations, so it should also be usable for anyone trying to use MTS from any programming language.

Approach?

It will be explained how to assign random frequencies to all midi notes by composing a valid sysex message containing bulk tuning information. The explanation is somewhat math heavy, but every step is completely detailed which should help you implement something in your favorite programming language.

MTS?

Midi tuning specification is a protocol that allows users of synthesizers to assign any frequency to any midi note. You can reverse your keyboard if you like, or you can assign random frequencies to all keys, or you can explore the wonderful world of microtonality and xenharmonics. The way to assign any frequency to any midi note is by sending midi sysex messages.

Sysex?

Sysex is a hook in the midi protocol that allows for sending anything over midi. "Anything" is to be interpreted as follows: "anything that fullfills the demands made on bytes sent over MIDI, i.e. the highest bit is always 0". This constraint means that in some cases data has to be encoded in a form that is suitable to be sent over midi, and this is also the case for MTS messages. We'll go over the calculations in excruciating detail in the rest of the article.

With some exceptions, sysex messages are not standardized. Any manufacturer can implement its own sysex messages. DSI, e.g. uses sysex to perform OS updates on their instruments. Such sysex messages typically are not publicly documented.

The sysex messages used for sending tuning information, on the other  hand, are standardized. They come in a few flavors. There's a message to request an synth to send over its tuning to a computer, but the DSI instruments appear not to respond to those messages, so I guess they are not supported.  There's another message to send tuning information from the computer to a synth, and that works well. Still other messages exist to retune a few notes in real-time, but I haven't tried those yet.

MTS message structure

MTS message structure is not very complicated, but it takes a bit of explanation. The full explanation can be read in the midi tuning standard spec, but it might contain some mistakes (currently under investigation).

First of all let's explain the general message layout. 

F0 F7 7E 7F 08 01 IDX C1 C2 .. C16 TA1_0 TA2_0 TA3_0 .. TA1_127 TA2_127 TA3_127 CS F7
  • F0 F7 appears at the start of any sysex message
  • 7E specifies that it's a non-real-time message.
  • 08 specifies that we're about to send tuning information
  • 01 specifies that we're about to send a bulk request, i.e. tuning information for all midi notes
  • IDX specifies a 0-based tuning index, this corresponds to the tuning index selected in DSI's global parameters menu, alternative tunings menu entry
  • C1 .. C16 are 7-bit ascii codes for the characters in the name of the custom tuning
  • TA1_0 TA2_0 TA3_0...TA1_127 TA2_127 TA3_127 are the actual tuning information. Tuning information for each midi note is described using three bytes, to be explained below.
  • CS is a checksum, also explained below
  • F7 signals the end of the sysex message

TA bytes tuning information structure

How do the three bytes TA1, TA2, TA3 encode tuning information for a midi note? Well, the first of the three bytes, TA1 is a number that says which lower midi note number is closest to the desired frequency, if the instrument would be tuned in 12-Tone equal temperament (i.e. freq  =  2(midinote-69)/12*440 Hz or vice versa midinote  =  12*log2(freq/440 Hz)) + 69, where midinote is a number 0-127, and freq is a float.

The bytes TA2 and TA3 then encode the difference in "cents" between the frequency indicated by the midi note number in TA1 and the desired frequency. The difference in cents between two frequencies f1 and f2 can be calculated as nc  =  1200*log2(f2/f1)), but supercollider provides some built-in functions to hide these formulas.

In the MTS message, 128 such triples TA1,TA2,TA3 are sent, one for each midi note. 

So if you wanted to retune the 50th midi note to the value of 670 Hz, you'd have to put as 50th triplet in the message the following bytes:

  • the closest, lower midi number corresponding to 670 would be 76 (midi note 76, on a 12-TET tuned instrument, corresponds to a frequency of  2(76-69)/12*440 = 659.26 Hz ). The way to calculate this 76 is by taking the floor of the formula that transforms frequency to midi note number: floor(12*log2(670/440 Hz) + 69) = 76. In hexadecimal, 76 is 4C. 
TA1_50 = 76 (or hex: 0x4C)
  • the difference in cents between 659.26Hz and 670Hz would then be 1200*log2(670/659.26) = 27.98908618457 cents. This difference in cents will always be < 100, because the reference frequency is the closest lower midi note (and the difference between two successive midi notes in 12 TET, is always exactly 100 cents).
  • We have 2 bytes TA2 and TA3 to encode this number 27.98908618457. Because in midi bytes, the high bit is reserved to indicate commands instead of data, that gives us a total of 14 bits (= 0 to 2^14-1) to encode a number in the interval [0, 100] (I include 100 because the difference in cents might e.g. be 99.995). In other words, you get a resolution of 100 cents / (2^14) = 0.0061 cents. That is what we call super high precision :) 
  • If you map number 27.98908618457 from the interval [0,100] to the interval [0, (2^14-1)], by means of the formula 27.98908618457 * (2^14-1)/100 ,you get 4585.4519896181, or after rounding 4585. This 4585 is the number that must be encoded in TA2 and TA3. This happens with bit masking operations:
TA2_50 = 4585 >> 7 = 35 (or hex: 0x23)
TA3_50 = 4585 & 127 = 105 (or hex: 0x69)

Checksum calculation

  • The checksum is calculated as a XOR between all the bytes in the message, except for the sysex header (F0 F7), the checksum itself (obviously) and the end of sysex byte at the end (7F).
  • Gotcha: at the end of the checksum calculation, be sure to "and" the result with 127 (=0x7F) to mask out the highest bit, otherwise you end up with an invalid MIDI message. This is not explicitly mentioned in the MTS standard, but it's implicitly assumed for all MIDI data bytes.

Gotchas

  • Beware of limited precision of floating point operations. In some cases, it may lead to unexpected results and invalid sysex messages as a result. I encountered such problems in my supercollider implementation of the tuning message and put in a workaround.
  • Be sure to check and double check that midi note numbers used as TA1 do not become negative or >127. Failing to do so will cause you hours of debugging (I should know by now!) to find out why sometimes - seemingly out of the blue - the synthesizer does unexpected and weird things when sending a new tuning (like randomly switching to a different program and changing some LFO settings) or why sending a new tuning doesn't seem to have any effect.
  • The MIDI MTS spec states that sending a tuning value with TA1, TA2, TA3 = 0x7F 0x7F 0x7F should be interpreted by the synth as "do not change tuning", but it seems that DSI instruments interpret it as "tune to the highest possible frequency" instead. For this reason the code currently just puts the standard note where no note should be present.

Supercollider code

Note that the following screenshot is part of a supercollider class, part of a larger system to communicate with the DSI prophet rev2 hardware synthesizer (the code can be found here: https://github.com/shimpe/sc-prophet-rev2 ), so you'd have to change some minor syntax details to get it working as a regular function (click the image to enlarge, or - even better - copy paste it from github (which may have a more recent version): https://github.com/shimpe/sc-prophet-rev2/blob/master/Classes/ScalaCalculator.sc).