How it Works

Nü's objective is to support the design of web-audio install (see video above), where the behavior of each client (device on the floor) is defined from Max. See it as a MiraWeb where the sound and processing happens on the connected devices.

Over-simplified concept:

Connect your laptop and a bunch of smartphones on the same WiFi network. On said laptop, launch the Nü server (see the getting started section) and connect the smartphones to the server. Open a new Max patch and instantiate a Nü object. Open the object help, randomly click around: sound should start coming out of the smartphones. Repeat.

The real thing:

A Nü session involves at least 3 components: a server, a client, and a Max patch. The server (a NodeJS server if you feel like digging) runs on a laptop, the Max patch as well (can even be the same), and the client on a smartphone connected to the web page the server serves. All three components thus need to be connected to the same network (WiFi, cellular, etc.) for them to address each other. The smartphone connected to the server becomes a client (also referred to as player): displaying a welcome screen, it stands ready for Max's instructions (relayed by the server).

These instructions are issues via OSC from a Max abstraction, referred to as a Nü module. Actually, a module designs a triplet [ Max-abstraction / server Class / client Class ]: messages from the abstraction are sent to the server, and re-directed to the concerned clients. Each module is designed to do one thing (and eventually do it right), much like Max's objects (sfplay, loadbang, gate, etc.). On the server and client sides, the messages sent by the Max abstraction are interpreted, most of the time triggering a dedicated method. The figure below details the architecture of a Nü session and the inter-contexts / intra-module communications.

Architecture of the Nü framework

Based on these modules, one can design a web-based performance without learning JavaScript / HTML / CSS. While a module on its own is limited to a unique feature, they can be combined to do any kind of passive or participative audio-visual rendering on the connected smartphones. The framework has also been designed to simplify the creation of new modules, if ever you feel it's missing a feature.

Getting Started

First, you need to install Max. Then, download the latest release of the Nü framework. Run the ./node/_install_<name-of-your-architecture> file in a terminal (only available for OSX at the moment). Example with OSX: 

 
                    # (adapt release number below) 

                    cd ~/Downloads/soundworks-nu-1.0 

                    ./node/_install_macos.sh

which should result in a (large) bunch of lines ending with: 

 
                    [soundworks-nu] public/js/player.js: successfully bundled 
 

                    ## installation complete ##

You then need to start the NodeJS server: from the ./node folder in a terminal window: 

 
                    npm run start

which should once more result in a bunch of lines, ending this time with: 

 
                    [HTTP SERVER] Server listening on http://127.0.0.1:8000

Open your favorite browser to the suggested url http://127.0.0.1:8000 to the Nü splash-screen (left below). Clicking on said screen will lead you directly to the “ready to play” screen (right). A smartphone does not truly becomes a functional client until it reaches the later.

Finally, add ./maxmsp/ and its subfolders to your Max path and open the ./maxmsp/extra/nu.getting-started.maxpat patcher. From there you should be able to produce a few notes and test your installation. If your smartphone is connected to the same network as the computer running both server and max patch, connect it to said server using its local IP address this time. Mine at the moment is 192.168.0.12, the resulting url: http://192.168.0.12:8000.

Start Designing

Every Nü module has its own Max abstraction, from which one controls both Client and Server parts of the module. As with standard Max objects, Nü abstractions can be combined to create complex scenarios (without having to meddle in JavaScript). The best way to start designing your own Max patch is probably to read through the content of the ./maxmsp/help folder, getting to know each module and its potential applications. The following video showcases some of these applications with the modules available as of mid 2017:

Always have a main at hand: Every Nü message in max is sent via an udpsend object to the Server. In Nü, this udpsend object is located in the ./maxmsp/patchers/nu.main.maxpat abstraction. This means that you need an instance of the nu.main.maxpat abstraction present in your Max patch at all time if you want messages to reach the Server. Alternatively, you can have an instance of the ./maxmsp/extra/nu.monitor-b-patcher.maxpat abstraction nested in a bpatch (it instantiates both a nu.main and a nu.output abstraction, the later quite useful for debug sessions, see below) .

Simulating devices on your laptop: Hopefully, the ./maxmsp/extra/nu.monitor-b-patcher.maxpat bpatcher coupled with the ./utils/openChromeArray.sh utility will help you figure out how your design will feel like once deployed in a concert hall on numerous devices. The latter is a bash script that opens a NxN array of Chrome windows that will emulate Nü clients connected to your server. The former is a debug tool that can be used for real-time auralization of the clients audio outputs (binaural + room impulse response): position a virtual listener in the midst of the emulated clients to ear them as you would in the real thing (or close enough to assess whether your design is heading the right way).

The Controller lives on a different webpage: If you stumble upon the ./maxmsp/nu.controller.maxpat module, try connecting your device to http://<server.ip.address>:8000/controller. This webpage is dedicated to your (as opposed to spectator's) devices, to e.g. control the behavior of your patch with touch, orientation, and acceleration sensors.

Development session layout. left: Max GUI, right: emulated Nü clients. An “audio beam” is projected in space (panned through the clients).

Go Full Scale

To deploy your newly designed web application and use it on the smartphones of (numerous, no doubt) spectators, you may need to modify the network configuration of the NodeJS server. Feel concerned by the following section only if you plan to use a WAN (as opposed to a LAN). Adapt the following parameters of the ./node/src/server/config/default.js to the deployment network:

development (original):
 
                    env: 'development' 

                    assetsDomain: '/' 

                    websockets: { 

                    ... 

                    path: '' 

                    }
production (deployment):
 
                    env: 'production' 

                    assetsDomain: 'http://my.domain.name.fr/myApp' 

                    websockets: { 

                    ... 

                    path: '/myApp/socket.io' 

                    }

If you intend to use Max on a different machine than the one running the NodeJS server, you'll also need to update the OSC field of the configuration file:

development (original):
 
                  // configuration of the `osc` service 

                  osc: { 

                    ... 

                    receiveAddress: '127.0.0.1', 

                    sendAddress: '127.0.0.1', 

                  }
production (deployment):
 
                  // configuration of the `osc` service 

                  osc: { 

                    ... 

                    receiveAddress: 'my.domain.name.fr', 

                    sendAddress: 'node.server.ip.address', 

                  }

And if you're planning to use secure connections (HTTPS rather than HTTP, may be required for streaming GPS data and the like), you'll need to generate your own key and certificate and indicate their path in the httpsInfos field.

Create New (Nü) Modules

Creating a new module will require to (1) learn JavaScript and (2) understand the way the framework has been designed. The former I leave to you. For the later, have a look at both nuTemplate and nuScore modules. Keep in mind that each module has 3 components:

  • Server class: ./node/src/server/NuMyModuleName.js
  • Client class: ./node/src/client/player/NuMyModuleName.js
  • Max patch: ./maxmsp/patchers/nu.myModuleName.maxpat
  • (optional) Max help patch: ./maxmsp/help/nu.myModuleName.maxhelp

From your readings, you should understand that:

  • All standard modules extend the NuBaseModule client/server Classes, responsible for the automatic routing of OSC messages to server and/or clients module methods and parameters, based on OSC headers.
  • If all players are concerned by an OSC message, it is stored in server's this.param attribute but if a method named after the OSC header is defined in the Server class of the module. this.param is broadcasted upon client connection to make sure every client shares a common state, even late comers.
  • Likewise, a Nü module on the Client side will save the values issued by its Server counterpart - values coming from OSC messages re-routed via web-socket by the server to concerned clients - to its this.param attribute but if it defines a method named after the OSC header.

Available modules

Name Description
Main Handle context synchronization between Max and Soundworks server
Group Playback of audio files on connected devices
Loop Distributed sequencer
RoomReverb Artificial distributed reverberation based on wave propagation simulation through the grid of connected devices
Path Draw audio trajectories through the grid
Synth Distributed synthesizer
Grain Distributed and synchronized granular synthesis
Stream Stream audio (sync) from Max to the connected devices
Output Device-less test of Nü setups (mock-up devices, simulating audio/visual, spatializing audio output based on defined device position comp. to the listener)
Probe Re-route devices sensors data to Max
Controller Same as probe, tailored for composer's devices
Display Control visual feedback on devices screens (e.g. level-meter)
Score Playback timed audio file sequences
Template Bare module

FAQ

How do I add my own audio samples to the available sounds?

Add your audio files to any existing / new folder of the ./node/public/sounds folder. The depth of the file tree does not matter as it's flattened at import. Restart the Nü server. Audio file names are referred to without their extension from Max, see e.g. ./maxmsp/extra/nu.getting-started.maxpat. Prefer .mp3 or .wav for you files extensions.

Opening nu.getting-started patch, Max console outputs a bunch of “nu.* : No such object”.

Add ./maxmsp/ and its subfolders to your Max path.

About

License:The Nü framework is released under the BSD 3-Clause license.

Credits: Nü has been developed at IRCAM-CNRS within the CoSiMa research project, supported by the French National Research Agency (ANR).