A dynamic signal flow graph implementation in max for live

Sourcecode on GitHub

Introduction

So after years i decided to open a blog and when i went to blogger.com i stumpled over the old blog i started years ago. Good luck for me so i don't need to create a new account and had a chance to view my old brain emisisons - did I write this?!

This time i wanna release my thoughts about creating a variable audio channel strip. Everything started when i wanted to have a better overview over my ableton live routings. I created this device. It consists of three samplers and a predefined signal chain where custom audio processing plugins can be loaded. The implementation is as stupid as it could be - i just copied the objects: Not a good exercise. A more dynamic approach would be nice. I researched a bit and stumpled over various advanced issues i wanna document here.

20200501: Graph meta definition

So the first issue when we wanna create a dyamic signal flow is to find a meta definition data structure to define the layout of the graph. Max provides the matrixctrl object that represents a 2d matrix consisting of rows and colums where the user can set dots on each cell. This is what i wanna use to define the signal flow graph, as such matrix controllers are common for mastering studios i think. Here they are called mastering patch bays or mastering matrix. A virtual version of this is what i wanna do.

20150501: Image: Signal flow matrix example with internal routings

So here are some definitions for our matrix:

Left side are inputs 1..n

top side are outputs 1..n

In1/Out1 is the main input/output of the device

The rest of the in/outs are internal routings

The internal output is connected to the appropriate internal input

So for n > 1 applies: out[n] is connected to in[n]

The object represented by one dot in the matrix we call node.

Each node in general consists of a audio processing plugin, 

Maybe other controls may be there like in the 24 subdevices of the Kickmachine device. 

This is why feedback loops may occur if the wrong routing is activated by setting the dot in the matrix. The first thing we wanna deal with is the detection of feedback loops. (See next chapter)

But back to the definition of the graph. It's as simple the graph is defined by a list of integer values (0, 1) which represents the activation state of all the nodes in the matrix. The index of a vlue is equal to the cellindex. Meaning first cellindex will be the dot on the top left, continuing from left to right and then line by line.

This layout is yet provided by the matrixctrls using getrow message for each row. We continue using this definition in the feedbackloopdetector patch. The only thing we have to do is request the single rows from the matrixctrl and join the lists together so we have one list which contains the value of the whole matrix in contrast to the values of a single row.

The result of the feedback detector may be several lists of cell-indexes which represent the paths of the feedback loop. But for the first step i did not implement this. Instead the algorithm gets triggered for one cell whichs enabled state shall be calculated and returns just a bool value 0,1 which represents the enabled state. The parent patcher triggers this routine for each cell in the matrix uppon changing a value in the matrix.

Gui: Matrix

The gui for the device shall make use of the matrixctrl object. Different modes for the matrix adjust the behaviour when clicking a button. The basic mode allready implemented in the prototype for feedback detection is to activate routings.

Others modes may exist so we have in total following modes:

Routing activation

Delete Plugin

Show Plugin Gui

Load plugin (from filesystem path)

Layer which shows green or red color for signal level clipping

20200501: Feedback Detection

By setting the matrix routings "wrong" you can produce feedback loops. Also in max this results in an awfull beeping noise in the audio output. And currently there is no detection of feedback loops. This is what i wanna do.

20150501: Image: Signal flow matrix example with feedback loop

You may have it guessed this is a recursive issue. A few hours should be enough to implement this on a programming language of my choice but i took the hard way and implemented it in max for live. The result of the feedback loop detector is used to disable cells to avoid the user configuring the matrix to contain feedback loops.

The prototype of the algorithm can be found here. It works pretty well but only inside the max editor since ableton prohibites the recursions and reacts with an stack overflow error.

.   20150501: Video: Signal flow matrix gui disables feedback loops.

20200511: CLR Adapter

After some investigation i decided to use the Max SDK and write a max object in C#. Following are some resources to get the C to C# Adapter running:

https://bountify.co/call-c-dll-from-c-application

NuGetPackageManager Command: Install-Package UnmanagedExports -Version 1.2.7

The sample project of the Max-C#-Adapter looks like this: Here are some inlets and outlets configured and some data repeated to the outlet.

namespace CbChannelStrip

{

using System.IO;
   using System.Drawing;

using CbMaxClrAdapter.Jitter;
   using CbMaxClrAdapter;


{
   public sealed class CChannelStrip : CMaxObject

this.IntInlet = new CIntInlet(this)
      public CChannelStrip()
      {
         {

this.IntOutlet = new CIntOutlet(this);
            Action = this.OnIntInlet
         };


Action = this.OnListInlet
         this.ListInlet = new CListInlet(this)
         {
         };

this.MatrixInlet = new CMatrixInlet(this)
         this.ListOutlet = new CListOutlet(this);

         {
            Action = this.OnMatrixInlet
         };

/// this.LeftInlet.SingleItemListEnabled = true; TODO-TestMe
         this.MatrixOutlet = new CMatrixOutlet(this);

         this.LeftInlet.Support(CMessageTypeEnum.Symbol);

this.LeftInlet.SetPrefixedListAction("load_image", this.OnLoadImage);
         this.LeftInlet.SetSymbolAction("clear_matrix", this.OnClearMatrix);
         this.LeftInlet.Support(CMessageTypeEnum.List);
      }

      internal readonly CIntInlet IntInlet;

internal readonly CMatrixInlet MatrixInlet;
      internal readonly CIntOutlet IntOutlet;

      internal readonly CListInlet ListInlet;
      internal readonly CListOutlet ListOutlet;

      internal readonly CMatrixOutlet MatrixOutlet;


this.MatrixOutlet.Message.Value.SetImage(Image.FromFile(aFileInfo.FullName)); 
      private void OnLoadImage(CInlet aInlet, string aSymbol, CReadonlyListData aParams)
      {
         var aFileInfo = new FileInfo(aParams.ElementAt(0).ToString().Replace("/", "\\"));
         this.MatrixOutlet.Message.Value.PrintMatrixInfo(this, "ImageMatrix");

this.IntOutlet.Message.Value = aInt.Value;
         this.MatrixOutlet.Send();
      }
       
      private void OnClearMatrix(CInlet aInlet, CSymbol aSymbol)
      {
         this.MatrixOutlet.Message.Value.Clear();
         this.MatrixOutlet.Send();
      }

      private void OnIntInlet(CInlet aInlet, CInt aInt)
      {

this.MatrixOutlet.Message.Value = aMatrix.Value;
         this.IntOutlet.Send();
      }

      private void OnListInlet(CInlet aInlet, CList aList)
      {
         this.ListOutlet.Message.Value = aList.Value;
         this.ListOutlet.Send();
      }

      private void OnMatrixInlet(CInlet aInlet, CMatrix aMatrix)
      {
         this.MatrixOutlet.Send();


      }
   }

}

This is the class diagram of the Adapter-Classes: (Download the image to view full size)

- Improvments not included herein: class MultiTypeOutlet

20200512: Gui: Graph

Another view would be a graph view we can create using graph wiz. With the image map generator of graph wiz and the apropriate image map object of max we can even create a limited interactive behaviour of the graph image.

In the meantime i made the graph view running in the version with the CLR-Adapter (see below). Here's a demo:

20200512: Video: Graph view visualizing the matrix state.

20200512: Latency Compensation

So each node can store one audio processing plugin. Depending on what the plugin does there may be no latency (for example a compressor) to a delay of several seconds (for example a phase linear eq) So when signals are processed in paralell and after the prcessing mixed together again there must be a strategy to compensate this latency. Otherwise it will result in unwanted phenomens starting from phasing problems up to a audible recognizeable delay in the signal. I put a sample signal flow here. This shall be the data for the first testcase. so the required output delay of a node will be: thiscomp = maxdelay - thisdelay. where max delay is the maximum delay of all paralell signal paths and this delay is the delay of the actual signal paths and thiscomp is the delay applied to the ringbuffer to realize latency compensation.

Here is the graph view with Latency Compensation. For the testcase i set Latency=IoNumber.

20200512: Video: Graphview with latency time displayed.

20200514: GraphView: Improved

I improved the graph view a bit so that Nodes which are not connected are displayed gray. Also Latency is not recognized for such nodes. Beside i changed the shapes for input, output and nodes. Also the scaling is improved.

20200512: Video: GraphView with improved layout.

20200514: GraphView: Animated (Beta)

I wrote some code to draw the graph via Vector2d API instead of dumping a bitmap. This allows to add some animation. Still the edges (the lines between the nodes) are missing. 

20200514: Video: Animated GraphView (Edges missing)

20200514: GraphView: Animated (Edges+Tips)

Here's the complete graph animated to morph between different states.

20200514: Video: Animated Graph View (With Edges+Tips)

20200515: GraphView: Animated (ColorMorph)

Here i added to morph colors, taking place when old color and new color differs.

20200515: Video: Animated Graph View (ColorMorph)

20200517: GraphView: Mouse & Keyboard Interaction

So i'm still was not satisfied with the user interface. So i removed the matrix view and added a drag & drop editor and keyboard commands. On top you see the final channel editor boxes with the buttons to open and show VST-Plugins. 

20200517: Video: Animated Graph View with Keyboard and MouseInteraction

20200518: Full-featured Gui Draft

Today i implemented the intermediatelly final layout for the gui and did a lot of improvements for the graph view. Behind the scenes the latency managment is nearly finished. No more long to go :)

20150518: Video: Full featured GUI

20150520: MixerMatrix with Latency Compensation running

So finally the latency compensation for vst plugins and the max device internal mc.send~ / mc.receive~ pairs is running. The plugin latency is received from the vst~ object and for the latency mc.send~ / mc.receive~we assume a hardcoded value of 1 sample per pair. 

All together for 11 channels we need 11 input buffers per channel and 11*11 + 2 send/receive objects. (2 for main in and main out). That results in 123 send/receive pairs. Actually we don't use that much since there are feedback loops which are not possible. But since max doesnt support dynamic alloction we need to place and allocate a send receive pair and a buffer for every location it can take place.

I turns out that max requires about 60-80 % cpu load and inside ableton live the cpu load is 150%. I still need to check this but it seems the cpu consumption comes from the send/receive pairs and the buffers.

So to sum that up, the pferormance is too weak. Maybe i gonna dig into multichannel audio buffer/mixer matrix programming and code that in c/c++. This should be faster since there is no send/receive object of max involved and everything can be solved in one set c/c++ stack frames without having max calls in between.

20150520: Image:First version with latency compensation and mixer matrix running.

20150522: Need for Optimization

A few days ago i put the max patcher into a ableton live max device and got shoked since the device which takes about 60% cpu load in the max editor uses 150% cpu load in max and hence is useless. A big discussion about possibilities for optimization followed here.

I allready started to set up a c++ project where i planed to implement the routing and latency compensation buffers on my own - without the overhead of max object and system calls in between. Surely implementing the buffers etc. in c++ would be a nice practice to get familar with audio processing but then another apporach came up, to connect the max objects dynamically at runtime.  So i'm not sure from device point of view it will be worth the effort to do a c++ buffer/routing implementation. I will decide in the next days which way to go. 

20150524: Optimization completed.

i extended the clr adapter with some classes that allow manipulating the patchers in c#. Using this mechanism the external for the mixer matrix now creates a hardwired max-graph that reflects the graph designed in the gui. By hardwiring the graph the matrix~, send~ and receive~ objects needed for dynamically switching the configuration of the routings are obsolete. Doing so the cpu load within max for live used for the graph routings goes down from 160% to nearly 0%. Now this is what we call a effective optimization... XD

20210201: Planned: C#-MaxObjectHost

It would be nice, to host a max object from c#, meaning not only create it in the patcher and wiring it but using it's inlets and outlets from c# code. This way i could extend the vst~ object with a unload function and also provide a native support for Shell Dlls.

20210201: Planned: C#-CodeGenerator

It would be nice, to analyze the max class registry and generate c# code from it so one can send specific messages to a concrete max object by calling a generated function.

20210201: Planned: Dynamic Resize Matrix

Add, remove nodes and resize matrix dynamically by keeping old connections.

20210201: Planned: CbVst~

Extended vst~ object supporting Unload and ShowShellDllPluginChooserDialog.