Magenta

Magenta Studio 2.0

Thu, 24 Aug 2023 07:00:01 -0700

TL;DR: Magenta Studio, first released in 2019, has been updated to more seamlessly integrate with Ableton Live. No functionality has changed, there are only UI changes and internal fixes. Please download and enjoy!

If you’re new to Magenta Studio, please read our previous post about what it is and how it works.

What’s New

In the previous version of Magenta Studio, the Max for Live (M4L) plugin would launch a separate application specific to your operating system for each of the tools. Unfortunately, as operating systems were upgraded, sometimes the applications stopped working. Therefore, we made the decision to integrate the tools directly into the Max for Live environment to ensure longer-term stability. The machine learning models are still directly integrated into the M4L plugin and do not require access to the Internet to use.

Upgrading

To upgrade from the previous version of Magenta Studio, you can download the latest version and drop it into Live directly in the place of the old plugin. The functionality has not been altered, only the interface and integration, so it works in exactly the same way.

Documentation

The documentation has been updated to reflect the new interface. The tool-specific videos have not been updated with the new interface, but the functionality is identical.

Support

Please report any issues to the GitHub repository. Thanks for using Magenta Studio!

Acknowledgements

Magenta Studio is based on work by members of the Google DeepMind team’s Magenta project along with contributors to the Magenta and Magenta.js libraries. The plug-ins were implemented by Yotam Mann and extended by Cassie Tarakajian.

The 2023 I/O Preshow – Composed by Dan Deacon (with some help from MusicLM)

Wed, 21 Jun 2023 13:00:00 -0700

Tl;dr: Dan Deacon worked with Google’s latest music AI models to compose the preshow music. Check out the MusicLM demo in the AI Test Kitchen app. Read on for more details about our collaboration with Dan Deacon.

Dan Deacon’s I/O Performance

On several occasions, we have had the pleasure of working with musicians that perform at Google I/O. This is an opportunity for us to bring our latest creative machine learning tools out of the lab and into the hands of the musicians. In previous years, we have worked with YACHT and The Flaming Lips. With YACHT we explored custom symbolic music generation models tailored to the band, and with The Flaming Lips we explored an interaction to bridge the audience and performers.

This year’s I/O pre-show was performed by electronic musician and composer Dan Deacon. With Dan we explored how artists might interact with generative models of music audio and incorporate them into their artistic process. Check out his performance in the video below and read on to learn more about his process using Google’s latest music AI tools:

Dan Deacon's performance at Google I/O 2023.

Dan used two of our new generative models in his performance: MusicLM (paper, demo), which produces music based on a text-based input prompt, and SingSong (paper), which will generate an accompaniment track for an audio-based singing input. Both of these models are part of the AudioLM (paper) family, and they directly produce audio based on the input conditioning (i.e., text or singing) by autoregressively predicting SoundStream (paper) tokens with one or more Transformer language models. SoundStream tokens can then be converted back to raw audio that can be used in conjunction with other audio editing software.

For his performance, Dan used MusicLM to create the chill, relaxing piano groove that’s heard behind his two meditations starring the Duck with Lips. Additionaly, Dan used both MusicLM and SingSong to create the Chiptune song. Most excitingly, Dan didn’t just use both SingSong and MusicLM, but actually extended their capabilities to put his performance together. We’ll discuss more of how Dan shaped the tools–and why it’s important that he did so–in the next section.

Working with Dan

As Dan discusses at around 7 minutes into his performance, he has always been excited by the promise that new technologies bring to the compositional process. Technology has a long and intertwined history with the art of making music. We might not think of things like flutes, violins, or trombones in the same way we think of computers now, but these were revolutionary new technologies when they were first introduced! They can also often seem disruptive at first–at one point in history, microphones caused quite a stir because they let vocalists sing much more softly (opposed to singing so loud they could be heard over the band). Yet in retrospect, microphones changed our relationship to music in many positive ways, enabling us to create, represent, and distribute music in ways that would have been inconceivable beforehand. Importantly, each new technological development expanded the creative palette of musicians, bringing with them new textures, new techniques, and sometimes new conceptions of music itself.

We view our new models as a continuation of music technology’s evolution. We’re incredibly inspired by the opportunity for these new tools to bring new creative capabilities to humanity, while remaining conscious of–and working hard to mitigate–their potential negative consequences. Our goal is and always has been to empower artists and musicians; a crucial piece of empowering musicians is understanding now these new tools situate themselves in different artists’ creative processes. With that in mind, collaborating with Dan was a great opportunity for us to work towards embodying our goals of empowering musicians in the era of generative modeling.

A glimpse of our in-person workshop where we showed our new tools to Dan Deacon.

About a month before I/O, we had a workshop with Dan where we introduced him to MusicLM and SingSong. Initially, Dan found many interesting text prompts to our MusicLM such as “a 600ft trombone.” He started to push the tools past their limit by, for example, playing his synthesizer into SingSong, ignoring that the system was trained on only singing inputs. These initial experiments turned out to be really fun and promising!

As we kept working with Dan, he surprised us by pushing these tools even further. Inspired by “I Am Sitting in a Room” (click here to listen), he fed the output of the SingSong model back into itself… over and over and over. Again, Dan moved beyond the model’s design of accepting singing input; by feeding its own output back into itself, the input audio was out of the distribution that the model had seen during training and we weren’t sure if this would work at all. Yet, not only did it work, but the feedback loop tended to produce music that still accompanies the input; it has the same key, tempo, and style. This was the interaction that Dan designed to compose the Chiptune song, above.

Dan began with a handful of text prompts to MusicLM, and then used the generated audio as input to SingSong and that output back through SingSong for numerous iterations. He was able to create hundreds of audio clips that complemented each other. From these, he handpicked his favorite clips, edited them slightly, and performed them.

We’re very proud to have been a part of Dan’s amazing performance. We’re extremely excited for the direction that this research is headed, and we’re always looking for ways to give musicians new tools to interact with. Check out the Google Keyword blog post to learn more about MusicLM and you can try it yourself by signing up via the AI Test Kitchen app.

Acknowledgements

This year’s I/O pre-show was a huge collaborative effort. We would like to thank everyone involved in making the performance a success (in no particular order): Josh Christman, Daniel Chandler, Meghan Reinhardt, Carolyne De Bellefeuille, Adi Goodrich, Jon Barron, Meghan Reinhardt, Carolyne De Bellefeuille, Irina Blok, Spencer Sterling, Ruben Beddeleem, Ben Poole, Cadie Desbiens-Desmeules, Chris Donahue, Jorge Gonzalez Mendez, Noah Constant, Jesse Engel, Timo Denk, Andrea Agostinelli, Neil Zeghidour, Christian Frank, Mauricio Zuluaga, Hema Manickavasagam, Tom Hume, and Lynn Cherry.

The Wordcraft Writers Workshop: Creative Co-Writing with AI

Thu, 01 Dec 2022 08:00:00 -0800

A core piece of Magenta’s mission is to empower creativity using AI and machine learning. In order to evaluate how well this goal is being achieved, it is important to put tools in the hands of creators, encouraging them to share honest and critical feedback. This feedback can help researchers to thoughtfully develop the next generations of ML-powered creative tools. Most of our prior efforts to engage with creators have been in the domain of music (for example, Magenta Studio and NSynth).

However, human creativity encompasses far more than just music: visual artists paint, draw, and sculpt, and writers craft stories and poetry. In recent years, we’ve seen huge advancements in machine learning techniques that can facilitate creativity in these other modalities. Creative writing is an especially interesting domain because it is so challenging for AI to get right. Even short stories commonly have narrative arcs that span paragraphs or longer, multiple characters with diverging points of view, and a careful balance of familiar archetypes and novel storytelling–all difficult traits for state-of-the-art AI to replicate. At the same time, the omnipresent writer’s block is not a problem at all for neural language models like LaMDA, which can effortlessly generate as many words as you ask them for.

Earlier this year, we invited a cohort of 13 professional creative writers to try their hands at writing stories using Wordcraft, an AI-augmented text editor with a wide range of generative capabilities targeted at creative writing assistance. Wordcraft can suggest story ideas, rewrite text according to user-provided instructions, and elaborate on what has already been written. It also has a chatbot interface where users can engage with LaMDA, Google’s dialog-based language model, about their stories.

A demo of the Wordcraft web application

As in generative music, AI-assisted story writing can be a mixed bag. At its best, Wordcraft made suggestions that were inspiring and surrealistic, and writers applauded its usefulness for ideation and overcoming writer’s block. However, it also had a tendency to rehash tired tropes, and it could take wading through many dull suggestions before finding an interesting one.

All of the writers’ stories are available in the Wordcraft Writer’s Workshop’s digital literary magazine, and a detailed writeup of what we learned about the role machine learning can play in creative writing can be found here.

We hope you enjoy perusing through the stories, and we are excited to hear your ideas about how AI can create valuable creative writing tools.

The Chamber Ensemble Generator and CocoChorales Dataset

Fri, 30 Sep 2022 09:00:00 -0700

In this post, we’re excited to introduce the Chamber Ensemble Generator, a system for generating realistic chamber ensemble performances, and the corresponding CocoChorales Dataset, which contains over 1,400 hours of audio mixes with corresponding source data and MIDI, multi-f₀, and per-note performance annotations.

🎵Audio Examples

📝arXiv Paper

📂Dataset Download Instructions

Github Code

Data is the bedrock that all machine learning systems are built upon. Historically, researchers applying machine learning to music have not had access to the same scale of data that other fields have. Whereas image and language machine learning researchers measure their datasets by the millions or billions of examples, music researchers feel extremely lucky if they can scrape together a few thousand examples for a given task.

Modern machine learning systems require large quantities of annotated data. With music systems, getting annotations for some tasks–like transcription or f₀ estimation–requires tedious work by expert musicians. When annotating a single example correctly is difficult, how can we annotate hundreds of thousands of examples to make enough data to train a machine learning system?

In this post, we introduce a new approach to solving these problems by using generative models to create large amounts of realistic-sounding, finely annotated, freely available music data. We combined two structured generative models–a note generation model, Coconet, and a notes-to-audio generative synthesis model, MIDI-DDSP–into a system we call the Chamber Ensemble Generator. As its name suggests, the Chamber Ensemble Generator (or CEG) can generate performances of chamber ensembles playing in the style of four-part Bach chorales. Listen to the following examples performed by the CEG:

String Ensemble Mixture:

Soprano: Violin 1	Alto: Violin 2

Tenor: Viola	Bass: Cello

Woodwind Ensemble Mixture:

Soprano: Flute	Alto: Oboe

Tenor: Clarinet	Bass: Bassoon

We then used the CEG to create a massive music dataset for machine learning systems. We call this dataset CocoChorales. What’s exciting about the CEG is that it uses a set of structured generative models which provide annotations for many music machine learning applications like automatic music transcription, multi-f₀ estimation, source separation, performance analysis, and more.

Below, we dig deeper into each of these projects.

The Chamber Ensemble Generator

As we mentioned, the Chamber Ensemble Generator (CEG) is a set of two structured generative models that work together to create new chamber ensemble performances of four-part chorales in the style of J.S. Bach.

As seen in the figure above, the constituent models in the CEG are two previous Magenta models: Coconet and MIDI-DDSP. Coconet is a generative model of notes, creating a set of four-instrument music pieces (“note sequences”), harmonized in the style of a Bach Chorale. Each of these four note sequences is then individually synthesized by MIDI-DDSP. MIDI-DDSP is a generative synthesis model that uses Differentiable Digital Signal Processing (DDSP) that turns note sequences into realistic audio that can sound like a number of different instruments (e.g., violin, bassoon, or french horn).

It’s important to note that the CEG is built on structured generative models, i.e., models that have interpretable intermediate representations. On the one hand, this structure leads to a very opinionated view of music. The CEG is limited in ways that other generative music models are not; it cannot generate all styles of music, like a rock and roll ensemble for example. It can only generate chorales. However, many generative music models are notoriously “black boxes,” whose internal structures are difficult to interpret. By being built on a modular set of structured models, the internals of the CEG are easy to understand and modify. This also allows us to create a dataset with many types of annotations that would be tedious or impossible to acquire with other types of generative models (such as annotations of the velocity and vibrato applied to each individual note in a performance). In the next section, we will showcase how these interpretable structures can be used to mitigate biases of these generative models.

The CocoChorales Dataset

CocoChorales is a dataset of 240,000 examples totalling over 1,400 hours of mixture data. The Chamber Ensemble Generator (CEG) was used to create CocoChorales by sampling from the CEG’s two constituent generative models, Coconet and MIDI-DDSP. Using the CEG in this way is an example of dataset “amplification,” whereby a generative model trained on a small dataset is used to produce a much larger dataset. In this case, we are amplifying two very small datasets: Coconet is trained on the J.S. Bach Chorales Dataset, which contains 382 examples, and MIDI-DDSP is trained on URMP, which contains only 44 examples. But, using the CEG, we were able to generate 240,000 examples!

CocoChorales has examples performed by 13 different instruments (violin, viola, cello, double bass, flute, oboe, clarinet, bassoon, saxophone, trumpet, french horn, trombone, and tuba) organized into 4 different types of ensembles: a string ensemble, a brass ensemble, a woodwind ensemble, and a random ensemble (see the CocoChorales dataset page for more info). Each example contains an audio mixture, audio for each source, aligned MIDI, instrument labels, fundamental frequency (f₀) for each instrument, notewise performance characteristics (e.g., vibrato, loudness, brightness etc of each note), and raw synthesis parameters.

What’s cool about using the structured models in the CEG, is that because the system is modular, it is easy to interpret the output of the intermediate steps of the internal CEG models. For example, the MIDI-DDSP model we used tended to produce performances that were oftentimes out of tune and skewed sharp (i.e., frequency of a note being played was often slightly higher than the “proper” tuned frequency of the note in the piece, according to a 12-TET scale). This is visualized by the orange histogram in the above image (labeled “w/o pitch aug”), which shows how in or out of tune each note is once every 4ms (here, 0.0 means perfectly “in tune”). We were able to correct for this systematic bias by directly adjusting the f₀ curves output by the synthesis generation module of the MIDI-DDSP model, as shown by the blue histogram (labeled “w/ pitch aug”), which shows a distribution that is more centered on 0.0 in the figure above. This level of control is hard to achieve with black box generative models, and a big reason why we’re very excited about using the structured models in the CEG.

Downloading the Dataset

We’re really excited to see what the research community can do with the CocoChorales dataset. Further details on the dataset can be found here. Instructions on how to download the dataset can be found at this Github link.

If you want to learn more about either project, please see our arXiv paper. The code for the Chamber Ensemble Generator is available here and usage instructions are here. If you use the Chamber Ensemble Generator or CocoChorales in a research publication, we kindly ask that you use the following bibtex entry to cite it:

@article{wu2022chamber,
  title = {The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling},
  author = {Wu, Yusong and Gardner, Josh and Manilow, Ethan and Simon, Ian and Hawthorne, Curtis and Engel, Jesse},
  journal={arXiv preprint arXiv:2209.14458},
  year = {2022},
}

Autoregressive long-context music generation with Perceiver AR

Thu, 16 Jun 2022 09:00:00 -0700

We present our work on music generation with Perceiver AR, an autoregressive architecture that is able to generate high-quality samples as long as 65k tokens—the equivalent of minutes of music, or entire pieces!

🎵Music Samples

📝ICML Paper

GitHub Code

DeepMind Blog

The playlist above contains samples generated by a Perceiver AR model trained on 10,000 hours of symbolic piano music (and synthesized with Fluidsynth).

Introduction

Transformer-based architectures have been recently used to generate outputs from various modalities—text, images, music—in an autoregressive fashion. However, their compute requirements scale poorly with the input size, which makes modeling very long sequences computationally infeasible. This severely limits models’ abilities in settings where long-range context is useful for capturing domain-specific properties. Music domains offer a perfect testbed, since they often exhibit long-term dependencies, repeating sequences and overall coherence over entire minutes—all necessary ingredients for producing realistic samples that are pleasing to the human ear!

To ameliorate these issues, we propose Perceiver AR, an autoregressive version of the original Perceiver architecture. A Perceiver model maps the input to a fixed-size latent space, where all further processing takes place. This enables scaling up to inputs of over 100k tokens! Perceiver AR builds on the initial Perceiver architecture by adding causal masking. This allows us to autoregressively generate music samples of high quality and end-to-end consistency, additionally achieving state-of-the-art performance on the MAESTRO dataset.

Setup

Perceiver AR first maps the inputs (in the diagram, [P,e,r,c,e,i,v,e,r,A,R]) to a fixed-size latent array, via a single cross-attention operation. These latents (3 illustrated above) then interact in a deep stack of self-attention layers to produce estimates for each target. The most recent inputs ([r,A,R]) correspond to queries, and each latent corresponds to a different target position ({1: A, 2: R, 3: <EOS>}).

Causal masking is used in both kinds of attention operations, to maintain end-to-end autoregressive ordering. Each latent can therefore only attend to (a) itself and (b) latents corresponding to ‘earlier’ information (either input tokens or target positions). This respects the standard autoregressive formulation, where the probability distribution for the t-th output is only conditioned on what was generated at previous timesteps 1, ..., t-1.

In the music domain, we use up to 65k-token inputs, which corresponds to several minutes in the symbolic domain and one minute in the raw audio domain.

Symbolic music

The playlist at the top showcases 8 unconditional samples. These were generated by a model that was trained on 10,000 hours of transcribed YouTube piano performances containing examples between 1k and 32k tokens in length. The model had 1024 latents and 24 self-attention layers. Training on this large-scale dataset yields high-quality samples with stylistic and structural coherence—one can identify repeating musical themes, different chord progressions, arpeggios and even ritardandos. Moreover, the main difference from our previous model trained on YouTube piano performances is that a 32k input size was feasible this time, so we only used full-length pieces for training! This allowed Perceiver AR to better model entire pieces with beginning, middle and end sections.

Next, we present audio samples from the symbolic domain, obtained by training on MAESTRO v3. The input representation in both cases was computed from MIDI files as described by Huang et al. in Section A.2, and the final outputs were synthesized using Fluidsynth.

Raw audio

Perceiver AR can also be used to generate samples from the raw audio domain. Here, we applied the SoundStream codec to MAESTRO v3 .wav files to encode the raw audio. After training the model, we generated samples and decoded them into the source domain. Keeping the context length fixed, we experimented with 3 different codec bitrates—12kbps, 18kbps, 22kbps—which, for an input length of 65k tokens, span 54.4s, 36.8s and 29.6s of music, respectively. The examples below illustrate the trade-off between sample duration and fidelity: codecs with lower bitrates model coarser structure and enable training on a longer period of time, but sacrifice audio quality.

12kbps

18kbps

22kbps

You can listen to more raw audio samples here 🎵.

Bonus

To end on a high note (🙃), we invite you to enjoy Charlie Chen’s creation - a music box that plays Perceiver AR outputs, adding an immensely nostalgic feel to the generated music!

@inproceedings{
  hawthorne2022general,
  title={General-purpose, long-context autoregressive modeling with Perceiver AR},
  author={Hawthorne, Curtis and Jaegle, Andrew and Cangea, C{\u{a}}t{\u{a}}lina and Borgeaud, Sebastian and Nash, Charlie and Malinowski, Mateusz and Dieleman, Sander and Vinyals, Oriol and Botvinick, Matthew and Simon, Ian and others},
  booktitle={The Thirty-ninth International Conference on Machine Learning},
  year={2022},
  url={https://arxiv.org/abs/2202.07765}
}

DDSP-VST: Neural Audio Synthesis for All

Wed, 08 Jun 2022 13:00:00 -0700

🎵Get the plugin

Train your own model

Introduction

Back in 2020, we introduced DDSP as a new approach to realistic neural audio synthesis of musical instruments that combines the efficiency and interpretability of classical DSP elements (such as filters, oscillators, reverberation, etc.) with the expressivity of deep learning. Since then, we’ve been able to leverage DDSP’s efficiency to power a variety of educational and creative web experiences, such as Tone Transfer, Sounds of India, and Paint with Music. However, there’s one question we’ve received more than any other: “Nice! When can I get that plugin?”

We’re happy to announce that finally, now you can! Introducing DDSP-VST, a cross-platform real-time neural synthesizer and audio effect that you can run directly in your favorite digital audio workstation.

What can you do with DDSP-VST?

Transform your voice or other sounds into a variety of musical instruments in effects mode
Play neural synthesizers with MIDI just like a typical virtual instrument
Explore wild and funky new timbres with intuitive tone shaping controls
Train your own models for free with our web trainer, and share with your friends!
Contribute your own cool features to the plugin (it’s open source!)

Why?

Our goal with Magenta has always been to go beyond the research papers to create tools that help artists and musicians explore the future of Machine Learning and Creativity. The research into DDSP was inspired by the challenge of making neural audio synthesis more tangible and accessible for everyone.

With DDSP-VST we hope to make it easy for anyone to not only use neural synthesizers in their music, but to also empower anyone to “get behind the steering wheel” and train and share their own models. Machine learning is infused with the bias of those who get to train the models, and we want to open that opportunity to more people to have their voices heard.

Using our accompanying web trainer, anyone can train a model for free with only a few minutes of audio training data.

How does it work?

DDSP-VST uses the same algorithm and operating principles as other DDSP applications before it, with some slight adjustments to improve efficiency for low-latency real-time audio. There are 3 stages to the plugin, each with their own unique set of controls

Feature Extraction
Predict DSP Controls
Synthesis

Feature Extraction

The key to the plugin is that it represents incoming audio as just two signals: Pitch and Volume. The volume is just the RMS value of the incoming sound, and the pitch is extracted using a trained neural network.

The original paper used a CREPE model for good accuracy, and our web application used a SPICE model for more efficiency. For the plugin, we pushed the efficiency even further by using network distillation to train a new tiny CREPE model that is close to the accuracy of the original with just ~160k parameters (137x less).

Predict DSP Controls

The second neural network predicts the controls for an additive harmonic synthesizer and a subtractive noise synthesizer for every pitch and loudness it receives. For efficiency, we trained a much smaller recurrent neural network (RNN) than the original paper, and found we could increase the time between predictions from 4ms to 20ms without much change in audio quality for a further 5x boost in efficiency. We also had to pay special attention to not incorporate any non-causal convolutions like the web versions of DDSP, as that would create latency from the model trying to “see into the future”.

Synthesis

The outputs of these DSP synthesizers are then mixed together to produce the final audio output. In this way the RNN can take the volume and pitch from any input sound it wasn’t trained on and play the sound of the instrument it was trained on with the same volume and pitch contours.

Tone Shaping

But, hold on… not all instruments can play all pitches. Flutes play high, basses play low–so, what happens if we feed in sounds with low pitches for a model trained on high pitches? We get some fun and wild new timbres!

Enter “Tone shaping” , a control that allows us to easily explore this space of sounds (and also get realistic sounds if that’s what we’re after). We can see our current volume and pitch by the bouncing purple ball. The range of volumes and pitches the model was trained on are shown by the highlighted box. When the ball is inside the box, we’re giving inputs like the training data so we get more realistic timbres. When it’s outside the box, we hear how the RNN predictions break down in fun and creative ways for values outside of training.

By allowing the box to be movable in both pitch and volume, we can shape the tone, to get wild variations of timbre, or get realistic tones by moving it to cover the ball. For the most realistic results, use the Pitch Shift knob to shift the input pitch into the original range of the box which is the original pitch range of the instrument (e.g. high for flutes, low for bass, etc.)

MIDI Synthesizer

We have also created a MIDI instrument version of the plugin, where pitch and volume contours are created directly from MIDI inputs. This enables using sequences, keyboards, arpeggiators, and other MIDI sources/effects to drive the plugin, and for neural synthesis to fit into existing workflows. It also avoids any pitch detection errors from the tiny CREPE model, for a more polished sound.

Train your own

One of our guiding goals has always been to democratize machine learning for artists and musicians, empowering creatives to decide for themselves how to use this new technology in their creative process. This allows for more diversity and control from practitioners and less constraints and bias from researchers.

One of the biggest barriers has always been allowing creatives to train their own models, as the training process usually requires a lot of training data and computational power. DDSP overcomes these challenges with the built-in structure of the model. This enables anyone to train their own model with as little as a few minutes of audio and a couple hours on a free Colab GPU. We’ve even observed that training can be finished in less than an hour on Colab Pro accounts.

Try out the free web trainer to train your own models and share with your friends.

Contribute

We think this plugin offers new opportunities to engage with a community of creators and we’re excited to see what everyone does with it. Creating the plugin was a community effort, and we look forward to improving it together. If you’re a musician, please try it out and share your models and music and feedback. The code is also open source for those willing and able to improve the plugin itself. We hope this project can also serve as a helpful starting point for others looking to make their own plugins with embedded ML models.

Acknowledgements

This has truly been a “passion project” with many people contributing their free time and advice to help make it happen. We would especially like to thank Holly Herndon, Matt Dryhurst, Hannes Widmoser, Rigel Swavely, Chet Gnegy, and Georgi Marinov for all of their contributions, effort, and advice.

MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

Thu, 20 Jan 2022 08:00:00 -0800

We are pleased to introduce MIDI-DDSP, an audio generation model that generates audio in a 3-level hierarchy (Notes, Performance, Synthesis) with detailed control at each level.

Colab Demo

🤗Spaces

🎵Audio Examples

📝ICLR Paper

GitHub Code

💻Shell Utility

MIDI is a widely used digital music standard for creating music in live performances or recordings. It allows us to use notes and control signals to play synthesizers and samplers, for instance sending “note-on” and “note-off” information when pressing a key of a MIDI piano keyboard. While synthesizers can produce many sounds for each note, it is difficult to find control settings to produce realistic-sounding audio. On the other hand, samplers and black-box neural audio synthesis (e.g., WaveNet) can sound more realistic but offer less individual control.

Recently, we proposed DDSP, which combines DSP and neural networks to generate realistic audio with interpretable controls. While it’s fun to manipulate individual amplitudes and frequencies, these features are very low-level (250 times a second), so we wanted a way to also play DDSP models with more familiar note-level controls like MIDI.

In this blog post, we introduce MIDI-DDSP, which leverages the expressive power of DDSP to create realistic-sounding audio, but also enables editing from slower input data like MIDI.

Specifically, MIDI-DDSP expands the DDSP synthesis into three levels: Note, Performance, and Synthesis, as shown in the figure above. These levels are designed to be analogous to how humans create music. Given a note sequence, we use an expression generator to predict the note-wise expression controls of each note. Then, we use a synthesis generator to predict per-frame synthesis parameters from note sequence and note expression controls. Last, DDSP synthesizes audio waveform from synthesis parameters. Each level is interpretable and able to be exposed to the user, such that they can make edits and change the output audio at the Note level, the Performance level, and the Synthesis level. See the figure below for an example of this.

Fine-grained Control

Automatic Generation		Adjusted by Human Expert

We define six note expression controls for each note. You can change the dynamics, pitch, and timbre of each note in a performance by adjusting those expression controls to craft your version of the musical piece.

Hierarchical Generation

MIDI-DDSP can take input from different sources (human or other models) by designing explicit latent representations at each level. A full hierarchical generative model for music can be constructed by connecting MIDI-DDSP with an automatic composition model. Here, we show MIDI-DDSP taking note input from CoCoCo, a score-level Bach composition interface (powered by Coconet, the ML model behind the Bach Doodle), and automatically synthesizing a Bach quartet by generating explicit latent for each level in the hierarchy.

Full end-to-end generation with Coconet (Mix)

Soprano: Flute	Alto: Oboe

Tenor: Clarinet	Bass: Bassoon

More Audio Samples

Here we present more audio samples that are automatically generated from MIDI-DDSP, some with human adjustments for refinement.

Automatic Generation

Game of Thrones - Cello	BWV227.1 - String Quartet

Human Adjustment - Pirates of the Caribbean

Automatic Generation	Human Adjustment

You can listen to more audio examples here 🎵.

We hope you have fun with MIDI-DDSP and you can try it yourself to render MIDI, and even adjust note expressions or synthesis parameters! We have open-sourced the code, and we provided a Colab notebook for MIDI synthesis and interactive control. If you want to synthesize MIDI files in command-lines just like using tools like FluidSynth, we also provide a command-line interface for synthesis.

@inproceedings{
  wu2022midi,
  title={MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling},
  author={Yusong Wu and Ethan Manilow and Yi Deng and Rigel Swavely and Kyle Kastner and Tim Cooijmans and Aaron Courville and Cheng-Zhi Anna Huang and Jesse Engel},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/pdf?id=UseMOjWENv}
}

Many thanks to AK for creating a 🤗 Space for the project!

Paint With Music

Thu, 06 Jan 2022 20:00:00 -0800

Editorial Note: Here we present a blog post from our friends at Google Arts & Culture, who built a fun musical experiment based on DDSP.

In June 2021, the Google Arts & Culture Lab released an online experiment called Paint With Music, in which the users could create music by painting on sensorial canvases. This experiment is the final outcome of an adventure that started months prior, with a one-week sprint that gathered the Magenta team together with the Lab. This article is a brief summary about the making of this experiment.

Music for everyone

The initial goal of Paint With Music was to make music accessible and fun for musicians and non-musicians alike. With that idea in mind, Paint With Music connects music with another major form of artistic expression: painting. Helped by Magenta’s DDSP library (Differentiable Digital Signal Processing), the drawings of the user are translated into musical notes, performed by a chosen instrument. To make the experience more unique and personalized, the user can paint on sensorial canvases, such as the sky, or the ocean, amongst others.

A glitch snippet as a starting point

The starting point of this project was an open-source prototype made by two Magenta researchers, Jesse Engel and Ian Simon. The prototype, called ddsp-drawing, allowed the user to draw a line on a canvas, and the Y position of the line would be translated into a pitch. High notes were obtained by drawing on the top of the canvas, whereas low notes were obtained by drawing towards the bottom. Paint With Music is built on top of this prototype, and it uses the same principle; except it has more whales and birds!

The models

The power of Magenta is that it comes with great examples, snippets, and even pre-trained models that are ready to use! With that collaboration, the Google Arts & Culture team could use Magenta DDSP’s pre-trained models for 4 different instruments: violin, saxophone, trumpet and flute.

A canvas, some birds, and voilà!

To flesh out the experiment, some additional sounds and graphical elements were also added; elements of nature, such as whales and birds sounds. The graphics of Paint With Music are generated in canvas and are meant to replicate natural elements. The goal of these atmospheres was to render the experiment very easily understandable and intuitive for everyone. To enrich the sounds generated by DDSP, the javascript library tone.js was used.

Closing thoughts

Paint With Music was released in Summer 2021 and it has been very interesting to see what users did draw with it! Below are some of our favourite pieces shared by users.

The DDSP library holds a strong potential for experiments that tie the audio to the visual. Paint With Music is one interpretation of this principle, but it could be forever expanded and improved! For example, what if the user could draw with many strokes at once and create chords? What if they could change the strokes to create reverb or vibrato? The possibilities are endless and it is definitely a very fun and exciting field to explore!

Some examples of creations

Here is a gallery of examples made with our experiment. Enjoy!

Paint with music examples

Special thanks: Paint With Music would not have been possible without Jesse Engel, Ian Simon, Ben Tan, Edwin Toh, Asheley Gao, Bastien Girshig, and Aurora Straton.

HCI and ML: Putting People First

Wed, 15 Dec 2021 08:00:00 -0800

The goal of the Magenta project is not just to build powerful generative models, but to use those models to empower people to realize their creative goals. In seeking to optimize our models for human values, instead of just imitating datasets, we collaborate closely with Human-Computer Interaction (HCI) researchers (such as our great collaborators in People + AI Research (PAIR)) to iterate on our models, develop new interfaces for control, and find new ways to evaluate the true impact of our work.

We’re excited about the potential of such tight-knit collaboration between machine learning (ML) and HCI researchers to create better models and interfaces that empower people using machine learning for creativity. This post highlights a collection of this work that investigates how to promote creative ownership when co-creating with AI, express and communicate emotion through AI generated music, and use AI to mediate human-to-human collaboration.

🎵Create Music with COCOCO

📝COCOCO Paper

📝Expressive Communication Paper

📝Human-Human Collaboration Paper

Promoting Ownership in Human-AI Co-Creation

Recent generative music models have made it possible for anyone, regardless of their musical experience, to compose a song in partnership with AI. For example, the “Bach Doodle” was Google’s first-ever ML powered Doodle. By providing only a few notes, users could co-create a four-part composition with the AI in the style of J.S. Bach. This enabled anyone on the web to make new music in their browser. After the release of the Bach Doodle, we became curious how novice composers engage in co-creation activities like these, and how the design of the interactive tools could impact whether they feel personally-empowered when making music with generative models. In a user study with novice composers, we found that users can struggle to evaluate or edit the music when ML models generate too much content at once. Others wanted to go beyond randomly “rolling dice” to generate a desired sound, and sought ways to control the generation based on relevant musical objectives.

To improve the co-creation partnership, we developed COCOCO, a music editor web-interface that includes a set of AI Steering Tools that allowed users to restrict generated notes to particular voices and “nudge” outputs in semantically-meaningful directions. For example, a user could choose to generate a single accompaniment in the bass voice, and steer the generation to sound more sad.

Watch how a user might use COCOCO's interface for steering the generated music.

In a comparison study, we found that these new tools helped users increase their control, creative ownership, and sense of collaboration with the generative ML model. By observing and asking questions about novices’ strategies when composing, we found that the tools allowed people to compose the song bit-by-bit, and promoted controlling and understanding each part of the piece more than if the AI generated it all at once. This study showed that providing interfaces to partition and constrain the generation of an existing ML model can make a significant difference in composers’ creative experience and their partnership with the AI.

We invite you to make music using COCOCO!

If you’d like to use or extend this work, please cite our full paper published at CHI’20.

@inproceedings{louie2020novice,
  title={Novice-AI music co-creation via AI-steering tools for deep generative models},
  author={Louie, Ryan and Coenen, Andy and Huang, Cheng Zhi and Terry, Michael and Cai, Carrie J},
  booktitle={Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
  year={2020},
  url = {https://doi.org/10.1145/3313831.3376739}
}

Expressing and Communicating Emotion in Co-created Music

ML models are becoming more expressive and capable of generating music with long-range coherence. At the same time, better HCI interfaces for controlling them can promote feelings of ownership, as we saw with our previous project on COCOCO. While these parallel efforts are aimed at empowering the end-user, less is known about how ML models and HCI interfaces can impact a creator’s subjective experience, and how people objectively perform on a creative task, such as composing music to express an emotion.

The Expressive Communication framework brings together composers and listeners in comparing how well emotion is communicated through music made with different generative tools.

We created a unified framework called Expressive Communication to jointly evaluate the impact of ML and HCI advances in empowering expression. For the creative task, composers use different versions of a generative AI tool to create music with the goal of communicating a particular image and human emotion. This was inspired by real-world music creation tasks, such as making background music for a video, or creating music to set the mood for a scene in a film or video game. Additionally, the framework provides an objective measurement by using an outside listener to judge how the created music better evokes the intended imagery and emotions. We used this framework to compare two generative models capable of different degrees of long-range structure and musical coherence, and two different interfaces capable of different degrees of steering and iterative composition.

Our results show that both the ML and HCI approaches (developing better pretrained models and better steering interfaces, respectively) are important and complementary ways to support composers in both communicating through music and feeling empowered in the process of co-creating with generative models. Our results also shed light on how objectives of a pretrained model, such as stronger coherence, can make certain emotions such as “fear” more difficult to express with curation of random samples alone, and how the addition of steering interfaces can help to mitigate model biases by creating samples that are less likely from the model, but more aligned with the user’s expression and musical goals.

If you’d like to use or extend this work, please cite our recent paper.

@misc{louie2021expressive,
  title={Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces},
  author={Louie, Ryan and Engel, Jesse and Huang, Anna},
  year={2021},
  eprint={2111.14951},
  archivePrefix={arXiv},
  primaryClass={cs.HC}
  url={https://arxiv.org/abs/2111.14951}
}

Listen to the music and guess which generative tool was used

We created an interactive game for you to engage more as a reader and listener! We ask you to listen to two pieces of music created for the same emotion, and then guess which generative tool was used to make the compositions below.

Remember that more expressive models and more steerable interfaces led to more emotionally evocative and musically coherent pieces. Use this insight when guessing!

Which music was made with the more vs. less expressive model?

Music 1	Music 2

MusicTransformer (More Expressive)	PerformanceRNN (Less Expressive)

Reveal the answer

Which music was made with the more vs. less steerable interface?

Music 1	Music 2

Radio Interface (Less Steerable)	Chunks Interface (More Steerable)

Reveal the answer

Mediating Human-Human Collaboration with AI

Although there’s been a lot of recent research on human-AI collaboration, less is known about how generative ML tools, like the ones we are developing at Magenta, could affect human-human collaboration. Our HCI collaborators at Google studied the social dynamics of pairs of people composing music together, with and without the help of a generative music model.

Generative AI tools can help to provide a psychological safety net and mitigate interpersonal friction amongst human teams of composers, but can also reduce the depth of human collaboration.

We found that generative AI models can act as a “social glue” in co-creative activities — for example, generative AI helped human collaborators maintain forward momentum and cordiality in moments when there were creative tensions or disagreements. It also helped initially establish common ground between strangers, and served as a psychological safety net. Despite increasing the ease of collaboration, however, AI assistance may reduce the depth of human-human collaboration. Rather than grappling with each other’s ideas, users often offloaded that creative work to the generative AI. Users sometimes indicated that they felt more like joint “curators” or “producers” of art, rather than as the “composers” themselves. Researchers, designers, and practitioners should carefully consider these tradeoffs between ease of collaboration and depth of collaboration when building AI-powered systems.

If you’d like to use or reference this work, please cite our paper published at CHI’21.

@inproceedings{suh2021ai,
  title={AI as Social Glue: Uncovering the Roles of Deep Generative AI during Social Music Composition},
  author={Suh, Minhyang and Youngblom, Emily and Terry, Michael and Cai, Carrie J},
  booktitle={Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},
  year={2021},
  url = {https://doi.org/10.1145/3411764.3445219}
}

Conclusion

These projects on promoting ownership, supporting emotional communication, and exploring human-human music creation are just a taste of what is possible when people are put first during the entire lifecycle of developing and studying generative models. This is the reason to engage in these types of collaborations now, because the human needs and understanding of the impact on people-centered outcomes are equally important for the development of generative models.

We look forward to working in tandem with researchers to use cutting-edge HCI and ML research to create amazing experiences to delight and empower users. The initial interactive tools and experiences have been developed towards overcoming general challenges that any class of user, like a novice composer, could face. In the future, we hope that HCI + ML collaboration can lead to adaptive interfaces and models which best learn-from and support the personalized behaviors and needs of specific users.

Acknowledgements

Thank you to our paper co-authors for their significant contributions to the works we’ve covered here, including Andy Coenen, Minhyang Suh, and Emily Youngblom. A warm shout out to all the Googlers we have met during user studies: your curiosity and enthusiasm for music and using generative AI tools was energizing and much appreciated, and your thoughts and feedback has greatly impacted this work. Many thanks to the people across Google Research who have offered their help and guidance! Larger Magenta collaborators: Halley Young, Ethan Manilow, Josh Gardner, Sehmon Burnham, Ian Simon, Fjord Hawthorne, Pablo Samuel Castro, Riley Wong. PAIR Collaborators: Martin Wattenberg, Fernanda Viégas, Qian Yang, Sherry Yang, Emily Reif, Ellen Jiang, Alex Bäuerle; Fellow Research Interns: Sherry Tongshuang Wu, Katy Gero, Jialin Song, Guangzhi Sun. And many more!

Modern Evolution Strategies for Creativity

Thu, 18 Nov 2021 20:00:00 -0800

Editorial Note: We’re happy to have a featured post from our collaborator Yingtao Tian, a member of the Google Brain Tokyo team. Here he’ll discuss his most recent work on the intersection of non-differentiable programs and visual creativity. You can learn more about the project in the original paper and its online animated version.

Inspiration

Inspired by much of Magenta’s work that has primarily focused on the role of machine learning for music music, we recently began to wonder how similar approaches could inform froms of modern visual art, such as abstract painting. To provide better context to our approach, it’s helpful to have a brief review of the area.

Looking at painting art, starting from the early 20th-century in the wider context of Modernism, a series of avant-garde art has been advocating revolutionary, abstract point of views instead of traditional rules of perspective. These art movements, including Cubism, Geometric Abstraction, and Abstract Expressionism, have far-reaching impacts to Minimalist Art and Minimalist Architecture. Interestingly, such art forms have also been explored in computer arts, like Low-complexity Art, and Algorithmic Art, which, in a broader sense, also includes Genetic Algorithm where the artist determines the rules governing art generation. Such an approach has gained popularity over the years with the creative coding community, resulting in a number of sophisticated extensions.

With the Genetic Algorithm that helps artists in various forms as an inspiration, and especially the recent resurgence of interest in one of its branches, Evolution Strategies (ES), in the machine learning community, we choose to explore its usage for painting creativity applications.

Goals

We propose to synthesize painting by placing transparent triangles using Evolution Strategy (ES) to fit a concrete image or an abstract concept. The fitting process can be in some way controlled by the artists and multiple results can be presented for the artists to pick from.

For fitting the image with triangles, we explore modern ES algorithms. As shown in the figure below, we find it provides good quality and efficiency.

Figure: Our method fits the painting "Mona Lisa".
The target image (left) is followed by the evolution process that fits it (right)

We go one step further and explore fitting an abstract concept that is represented as a natural language sentence. Like what is shown in the following figure, we find it can produce diverse, distinct geometric abstractions that make sense when we consider how humans may interpret the language.


"Self"	"Walt Disney World"	"Google located at 1600 Amphitheatre Parkway in Mountain View, California."

Figure: Our method fits the concept represented as text. The concept is below the evolution process.

Finally and interestingly, the results produced by our method (to some degree) resemble Abstract Expressionism and Minimalist Art.

About the ES-based Creativity

One benefit we find with ES-based creativity is that our proposed method can fit any target image / concept and can handle a wide range of triangles due to the efficiency of the ES algorithm. We show that in two figures below. Also, the ES algorithm is capable of using the number of triangles as a “computational budget” where extra triangles could always be utilized for gaining fitness. This allows a human artist to use the number of triangles in order to find the right balance between abstractness and details in the produced art.

Target Image	10 Triangles	25 Triangles	50 Triangles	200 Triangles

Figure: Fitting several target images with different numbers of triangles. Images are "Darwin", "Mona Lisa" (both from Here), "Anime Face" (generated by Waifu Labs), "Landscape" (from Wikipedia), "Impressionism" (A May Morning in Moret by Alfred Sisley, compiled here).

	10 Triangles	25 Triangles	50 Triangles	200 Triangles
"Self"
"Human"
"Walt Disney World"
"A picture of Tokyo"
"Google located at 1600 Amphitheatre Parkway in Mountain View, California."
"The United States of America commonly known as the United States or America is a country primarily located in North America."

Figure: Fitting several abstract concepts with different numbers of triangles. The concept can be a single word ("Self" and "Human"), a phrase ("Walt Disney Land" and "A picture of Tokyo"), and a long sentence (last two examples).

Another benefit specific to abstract concept fitting is that our method is given much more freedom in arranging the configuration of triangles and can produce different solutions, as shown below. It is desired for computer-assisted art creation, since human creators can be put “in the loop”, not only poking around the text prompt but also picking from multiple candidates produced by our method.

	4 Individual Runs
"Self"
"Human"
"Walt Disney World"
"A picture of Tokyo"
"The corporate headquarters complex of Google located at 1600 Amphitheatre Parkway in Mountain View, California."
"The United States of America commonly known as the United States or America is a country primarily located in North America."

Figure: Fitting several abstract concepts with multiple runs with the same numbers of triangles.

Technical Details

The architecture of our proposed method is shown below. Basically, we represent a configuration of triangles in a parameter space which consists of positions and colors of triangles, render such configuration onto a canvas, and calculate its fitness based on how well the rendered canvas fits a target image or a concept in the form of a text prompt. The ES algorithm (we use PGPE with ClipUp optimizer) keeps a pool of candidate configurations and uses mutations to evolve better ones measured by the said fitness. When fitting a concrete image, we use the pixel-wise L2 loss between the canvas and the target image as the fitness; while for Fitting a concept, we first represent the concept as a text and embed the text prompt using the text encoder in CLIP, embed the rendered canvas using the image encoder also available in CLIP, and use the Cosine distance for for fitness.

Figure: The architecture of our method.

Closing words

It’s interesting to see that we can leverage ES to produce results with high quality, and produce geometric abstractions aligned with how humans perceive language and images. It also produces a distinct art style.

But this work in our opinion opens more questions than it answers — for example, maybe in the future, further investigation into the broader spectrum of art forms beyond the minimalism explored here should be conducted. Also, since ES is agnostic to the domain, i.e., how the renderer works, maybe in the future we will see ES-inspired approaches could potentially unify various domains with significantly less effort for adoption in the future.

How to Cite / Use this Work

The code for this work is open sourced and please let us know if you find it useful and/or build your work on top of it — we will be very happy to hear that!

If you use the proposed technique, please consider citing the paper where it was introduced:

Yingtao Tian and David Ha. "Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Conceptst." 2021. arXiv:2109.08857.

You can also use the following BibTeX entry:

@misc{tian2021modern,
    title={Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Conceptst},
    author={Yingtao Tian and David Ha},
    year={2021},
    eprint={2109.08857},
    archivePrefix={arXiv},
    primaryClass={cs.NE}
}