Machine studying is all over the place and the sphere of artwork isn’t any exception. This text explores functions of machine studying in portray and music.
Machine studying has emerged as an vital and highly effective instrument for software program builders. It has helped remedy issues that have been earlier thought of unsolvable, and has been adopted in a large spectrum of domains. Along with the assorted tech domains which have benefited from ML, many artwork varieties have additionally benefited from it. This text focuses on two main artwork varieties—music and portray.
Varied instruments can help in making use of ML to the inventive course of. A few of them are:
- Magenta
- Google Deep Dream
- MuseNet
There are numerous functions which have been developed utilizing widespread instruments resembling TensorFlow. This text has two main sections:
- The primary part explores neural fashion switch utilizing TensorFlow.
- The second part explores Magenta and its utility to music.
Neural fashion switch
Composing one picture within the fashion of one other picture is known as neural fashion switch. Utilizing this system, an strange image might be transformed into a mode of portray created by the masters resembling Picasso, Van Gogh, or Monet. The main steps in neural fashion switch are:
- Get a picture
- Get a reference picture
- Mix the pictures with ML
- Output the picture with the reworked fashion
Fig. 1 reveals a supply picture (the banner picture of an earlier article titled, ‘Is your web site accessible and inclusive?’ within the March 2020 situation of OSFY out there at
https://opensourceforu.com/2020/03/is-your-website-accessible-and-inclusive/) and a reference fashion picture (a Kandinsky portray).
The quick fashion switch utilizing TensorFlow Hub is proven in Fig. 2.
The code snippet with TensorFlow Hub is as follows:
import tensorflow_hub as hub
hub_module = hub.load(‘https://tfhub.dev/google/
magenta/arbitrary-image-stylization-v1-256/1’)
stylized_image = hub_module(tf.fixed(content_
picture), tf.fixed(style_image))[0]
tensor_to_image(stylized_image)
Within the fashion conversion course of, the next steps are concerned:
- Defining content material and elegance representations
- Constructing the mannequin
- Calculating the fashion
- Extracting the fashion and content material
- Working the gradient descent
- Complete variation loss
- Re-running the optimisation
The ultimate stylised output with the supply picture rendered utilizing the reference fashion picture is proven in Fig. 4.
The whole code demo is offered at
https://colab.analysis.google.com/github/tensorflow/docs/blob/grasp/web site/en/tutorials/generative/style_transfer.ipynb
MAGENTA
Magenta is an open supply venture aimed toward exploring the position of machine studying within the inventive course of (https://magenta.tensorflow.org/). There are two totally different distributions of Magenta, as listed beneath.
Magenta-Python. It is a Python library that’s constructed on TensorFlow. It has highly effective strategies to control music and picture information. Producing music by utilizing ML fashions is the first goal of Magenta-Python. It may be put in with pip.
Magenta-JavaScript. This distribution is an open supply API. It permits the usage of pre-trained Magenta fashions contained in the Net browser. TensorFlow.js is on the core of this API.
Magenta-Python
Step one is to put in Magenta together with the dependencies, as follows:
print(‘Putting in dependencies…’)
!apt-get replace -qq && apt-get set up -qq libfluidsynth1
fluid-soundfont-gm build-essential libasound2-dev libjack-dev
!pip set up -qU pyfluidsynth pretty_midi
!pip set up -qU magenta
Magenta basically works on NoteSequences, that are summary representations of a sequence of notes. For instance, the next code snippet represents ‘Twinkle Twinkle Little Star’:
from magenta.music.protobuf import music_pb2
twinkle_twinkle = music_pb2.NoteSequence()
# Add the notes to the sequence.
twinkle_twinkle.notes.add(pitch=60, start_time=0.0, end_time=0.5, velocity=80)
twinkle_twinkle.notes.add(pitch=60, start_time=0.5, end_time=1.0, velocity=80)
twinkle_twinkle.notes.add(pitch=67, start_time=1.0, end_time=1.5, velocity=80)
twinkle_twinkle.notes.add(pitch=67, start_time=1.5, end_time=2.0, velocity=80)
twinkle_twinkle.notes.add(pitch=69, start_time=2.0, end_time=2.5, velocity=80)
twinkle_twinkle.notes.add(pitch=69, start_time=2.5, end_time=3.0, velocity=80)
twinkle_twinkle.notes.add(pitch=67, start_time=3.0, end_time=4.0, velocity=80)
twinkle_twinkle.notes.add(pitch=65, start_time=4.0, end_time=4.5, velocity=80)
twinkle_twinkle.notes.add(pitch=65, start_time=4.5, end_time=5.0, velocity=80)
twinkle_twinkle.notes.add(pitch=64, start_time=5.0, end_time=5.5, velocity=80)
twinkle_twinkle.notes.add(pitch=64, start_time=5.5, end_time=6.0, velocity=80)
twinkle_twinkle.notes.add(pitch=62, start_time=6.0, end_time=6.5, velocity=80)
twinkle_twinkle.notes.add(pitch=62, start_time=6.5, end_time=7.0, velocity=80)
twinkle_twinkle.notes.add(pitch=60, start_time=7.0, end_time=8.0, velocity=80)
twinkle_twinkle.total_time = 8
twinkle_twinkle.tempos.add(qpm=60);
# It is a colab utility technique that visualizes a NoteSequence.
mm.plot_sequence(twinkle_twinkle)
# It is a colab utility technique that performs a NoteSequence.
mm.play_sequence(twinkle_twinkle,synth=mm.fluidsynth)
Machine studying with Magenta
The magenta.music library permits the developer to do the next:
- Create music utilizing utilities within the Magenta library by utilizing abstractions.
- Generate music utilizing ML fashions.
This library incorporates numerous ML fashions which can be constructed utilizing TensorFlow. Three of the distinguished fashions are listed beneath.
MelodyRNN. Given a word sequence, it would proceed within the unique fashion.
MusicVAE. That is used to construct novel NoteSequences. It can be used to interpolate between sequences.
Onsets and frames. This transcribes piano audio.
Melody RNN
Melody RNN is primarily an LSTM based mostly language mannequin. It handles musical notes. The target is to proceed the given sequence within the unique fashion. There are two steps to utilizing it:
- Initialising the mannequin
- Persevering with a sequence
The code snippet for mannequin initialising follows:
print(‘Downloading mannequin bundle. It will take lower than a minute…’)
mm.notebook_utils.download_bundle(‘basic_rnn.magazine’, ‘/content material/’)
# Import dependencies.
from magenta.fashions.melody_rnn import melody_rnn_sequence_generator
from magenta.fashions.shared import sequence_generator_bundle
from magenta.music.protobuf import generator_pb2
from magenta.music.protobuf import music_pb2
# Initialize the mannequin.
print(“Initializing Melody RNN…”)
bundle = sequence_generator_bundle.read_bundle_file(‘/content material/
basic_rnn.magazine’)
generator_map = melody_rnn_sequence_generator.get_generator_map()
melody_rnn = generator_map[‘basic_rnn’](checkpoint=None,
bundle=bundle)
melody_rnn.initialize()
The sequence continuation might be carried out with two parameters—the variety of steps and temperature. You possibly can tweak these two parameters and perceive the change within the resultant output:
# Mannequin choices. Change these to get totally different generated sequences!
input_sequence = twinkle_twinkle
num_steps = 128 # change this for shorter or longer sequences
temperature = 1.0 # the upper the temperature the extra random
the sequence.
# Set the beginning time to start on the following step after the final word ends.
last_end_time = (max(n.end_time for n in input_sequence.notes)
if input_sequence.notes else 0)
qpm = input_sequence.tempos[0].qpm
seconds_per_step = 60.0 / qpm / melody_rnn.steps_per_quarter
total_seconds = num_steps * seconds_per_step
generator_options = generator_pb2.GeneratorOptions()
generator_options.args[‘temperature’].float_value = temperature
generate_section = generator_options.generate_sections.add(
start_time=last_end_time + seconds_per_step,
end_time=total_seconds)
# Ask the mannequin to proceed the sequence.
sequence = melody_rnn.generate(input_sequence, generator_options)
mm.plot_sequence(sequence)
mm.play_sequence(sequence, synth=mm.fluidsynth)
Music VAE
Music VAE (variational auto encoder) is a generative mannequin that can be utilized to create novel sequences or interpolate between current sequences.
The mannequin initialisation might be performed with the next code snippet:
# Import dependencies.
from magenta.fashions.music_vae import configs
from magenta.fashions.music_vae.trained_model import TrainedModel
# Initialize the mannequin.
print(“Initializing Music VAE…”)
music_vae = TrainedModel(
configs.CONFIG_MAP[‘cat-mel_2bar_big’],
batch_size=4,
checkpoint_dir_or_path=’/content material/mel_2bar_big.ckpt’)
New sequence creation could also be carried out as proven beneath:
generated_sequences = music_vae.pattern(n=2, size=80,
temperature=1.0)
for ns in generated_sequences:
mm.plot_sequence(ns)
mm.play_sequence(ns, synth=mm.fluidsynth)
Magenta-JavaScript
As said earlier, the JavaScript distribution of Magenta can be utilized proper inside your Net browser. It supplies many highly effective options to control music within the Net browser.
The Magenta music might be integrated into JavaScript as follows:
Managing NoteSequences and enjoying them is somewhat easy in JavaScript:
participant = new mm.Participant();
participant.begin(TWINKLE_TWINKLE);
participant.cease();
The three machine studying fashions (MusicRNN, MusicVAE, and onsets and frames), which have been defined earlier with Magenta-Python, can be found within the JavaScript model as properly.
A pattern code snippet for MusicRNN is proven beneath:
// Initialize the mannequin.
music_rnn = new mm.MusicRNN(‘https://storage.googleapis.com/
magentadata/js/checkpoints/music_rnn/basic_rnn’);
music_rnn.initialize();
// Create a participant to play the sequence we’ll get from the mannequin.
rnnPlayer = new mm.Participant();
operate play() {
if (rnnPlayer.isPlaying()) {
rnnPlayer.cease();
return;
}
// The mannequin expects a quantized sequence, and ours was
unquantized:
const qns = mm.sequences.quantizeNoteSequence(ORIGINAL_
TWINKLE_TWINKLE, 4);
music_rnn
.continueSequence(qns, rnn_steps, rnn_temperature)
.then((pattern) => rnnPlayer.begin(pattern));
}
rnn_steps and rnn_temperature can be utilized to switch the output.
There are various demos out there at https://magenta.tensorflow.org/demos.
The Magenta Studio supplies a group of music plugins which can be constructed utilizing Magenta fashions (https://magenta.tensorflow.org/studio). The ability of Magenta might be understood from the number of demos.
With the arrival of superior libraries resembling Magenta, dealing with artwork with machine studying is changing into less complicated and more practical. We will certainly count on extra to return within the close to future.
To conclude, because the official documentation says, Magenta can actually make your browser sing.
This text was first printed in Might 2020 situation of Open Supply For You
Dr Ok.S. Kuppusamy is assistant professor of pc science, College of Engineering and Expertise, Pondicherry Central College. He has 14+ years of educating and analysis expertise in academia and trade. His analysis pursuits embrace accessible computing, Net data retrieval and cell computing
