Max (Cycling '74) leash on the WebAudio API: spectators devices as distributed speakers and sensors during live performances.
The objective of the Nü framework, based on Soundworks, is to give composers “control” over the speakers of spectators smartphones during a performance. Connecting their devices to a web page broadcasted by the performer's laptop, spectators become part of the composition: from simple sound sources to active musicians. The performer can then control the behaviors of Nü modules ‒ modular units (playback, distributed sequencer, etc.) designed for live composition ‒ via a series of Max objects. Nü has been developed as part of the CoSiMa Project.
Nü's objective is to support the design of web-audio install (see video above), where the behavior of each client (device on the floor) is defined from Max. See it as a MiraWeb where the sound and processing happens on the connected devices.
Connect your laptop and a bunch of smartphones on the same WiFi network. On said laptop, launch the Nü server (see the getting started section) and connect the smartphones to the server. Open a new Max patch and instantiate a Nü object. Open the object help, randomly click around: sound should start coming out of the smartphones. Repeat.
A Nü session involves at least 3 components: a server, a client, and a Max patch. The server (a NodeJS server if you feel like digging) runs on a laptop, the Max patch as well (can even be the same), and the client on a smartphone connected to the web page the server serves. All three components thus need to be connected to the same network (WiFi, cellular, etc.) for them to address each other. The smartphone connected to the server becomes a client (also referred to as player): displaying a welcome screen, it stands ready for Max's instructions (relayed by the server).
These instructions are issues via OSC from a Max abstraction, referred to as a Nü module. Actually, a module designs a triplet [ Max-abstraction / server Class / client Class ]: messages from the abstraction are sent to the server, and re-directed to the concerned clients. Each module is designed to do one thing (and eventually do it right), much like Max's objects (sfplay, loadbang, gate, etc.). On the server and client sides, the messages sent by the Max abstraction are interpreted, most of the time triggering a dedicated method. The figure below details the architecture of a Nü session and the inter-contexts / intra-module communications.
First, you need to install Max. Then, download the latest release of the Nü framework. Run the ./node/_install_<name-of-your-architecture> file in a terminal (only available for OSX at the moment). Example with OSX:
# (adapt release number below)
which should result in a (large) bunch of lines ending with:
[soundworks-nu] public/js/player.js: successfully bundled
## installation complete ##
You then need to start the NodeJS server: from the ./node folder in a terminal window:
npm run start
which should once more result in a bunch of lines, ending this time with:
[HTTP SERVER] Server listening on http://127.0.0.1:8000
Open your favorite browser to the suggested url
http://127.0.0.1:8000 to the Nü splash-screen (left below). Clicking on said screen will lead you directly to the “ready to play” screen (right). A smartphone does not truly becomes a functional client until it reaches the later.
./maxmsp/ and its subfolders to your Max path and open the
./maxmsp/extra/nu.getting-started.maxpat patcher. From there you should be able to produce a few notes and test your installation. If your smartphone is connected to the same network as the computer running both server and max patch, connect it to said server using its local IP address this time. Mine at the moment is
192.168.0.12, the resulting url:
./maxmsp/help folder, getting to know each module and its potential applications. The following video showcases some of these applications with the modules available as of mid 2017:
Always have a main at hand: Every Nü message in max is sent via an
udpsend object to the Server. In Nü, this
udpsend object is located in the
./maxmsp/patchers/nu.main.maxpat abstraction. This means that you need an instance of the
nu.main.maxpat abstraction present in your Max patch at all time if you want messages to reach the Server. Alternatively, you can have an instance of the
./maxmsp/extra/nu.monitor-b-patcher.maxpat abstraction nested in a bpatch (it instantiates both a
nu.main and a
nu.output abstraction, the later quite useful for debug sessions, see below) .
Simulating devices on your laptop: Hopefully, the
./maxmsp/extra/nu.monitor-b-patcher.maxpat bpatcher coupled with the
./utils/openChromeArray.sh utility will help you figure out how your design will feel like once deployed in a concert hall on numerous devices. The latter is a bash script that opens a NxN array of Chrome windows that will emulate Nü clients connected to your server. The former is a debug tool that can be used for real-time auralization of the clients audio outputs (binaural + room impulse response): position a virtual listener in the midst of the emulated clients to ear them as you would in the real thing (or close enough to assess whether your design is heading the right way).
The Controller lives on a different webpage: If you stumble upon the
./maxmsp/nu.controller.maxpat module, try connecting your device to
http://<server.ip.address>:8000/controller. This webpage is dedicated to your (as opposed to spectator's) devices, to e.g. control the behavior of your patch with touch, orientation, and acceleration sensors.
To deploy your newly designed web application and use it on the smartphones of (numerous, no doubt) spectators, you may need to modify the network configuration of the NodeJS server. Feel concerned by the following section only if you plan to use a WAN (as opposed to a LAN). Adapt the following parameters of the
./node/src/server/config/default.js to the deployment network:
If you intend to use Max on a different machine than the one running the NodeJS server, you'll also need to update the OSC field of the configuration file:
// configuration of the `osc` service
// configuration of the `osc` service
And if you're planning to use secure connections (HTTPS rather than HTTP, may be required for streaming GPS data and the like), you'll need to generate your own key and certificate and indicate their path in the
nuScore modules. Keep in mind that each module has 3 components:
From your readings, you should understand that:
NuBaseModule client/server Classes, responsible for the automatic routing of OSC messages to server and/or clients module methods and parameters, based on OSC headers.
this.param attribute but if a method named after the OSC header is defined in the Server class of the module.
this.param is broadcasted upon client connection to make sure every client shares a common state, even late comers.
this.param attribute but if it defines a method named after the OSC header.
|Handle context synchronization between Max and Soundworks server
|Playback of audio files on connected devices
|Artificial distributed reverberation based on wave propagation simulation through the grid of connected devices
|Draw audio trajectories through the grid
|Distributed and synchronized granular synthesis
|Stream audio (sync) from Max to the connected devices
|Device-less test of Nü setups (mock-up devices, simulating audio/visual, spatializing audio output based on defined device position comp. to the listener)
|Re-route devices sensors data to Max
|Same as probe, tailored for composer's devices
|Control visual feedback on devices screens (e.g. level-meter)
|Playback timed audio file sequences
How do I add my own audio samples to the available sounds?
Add your audio files to any existing / new folder of the
./node/public/sounds folder. The depth of the file tree does not matter as it's flattened at import. Restart the Nü server. Audio file names are referred to without their extension from Max, see e.g.
./maxmsp/extra/nu.getting-started.maxpat. Prefer .mp3 or .wav for you files extensions.
nu.getting-started patch, Max console outputs a bunch of “nu.* : No such object”.
./maxmsp/ and its subfolders to your Max path.