The Listening Machine: How It Works

At the core of The Listening Machine is a piece of software that monitors the Twitter activity of 500 people around the UK, selected proportionally from eight different fields: arts, business, education, health, politics, science, sport and technology. A further part of the group is selected completely at random. When any of these people posts a status update, the machine analyses its properties in terms of both sound and meaning, and generates music based upon it. Using techniques from the fields of natural language processing and machine learning, each tweet can be analysed from a number of perspectives:

  • Its sentiment: is it positive, negative or objective? Emotive words (“brilliant”, “rubbish”, “awesome”) and emoticons will rank as positive or negative accordingly, whereas technical terms (“pronunciation”, “structured”) rank as neutral, or objective.
  • Its classification: is it about a specific topic? The same set of 8 categories are used to classify each status update, using an archive of material from the BBC News website.
  • Its prosody, or rhythm of speech and intonation: vowel sounds and rhythmic patterns are extracted to translate sequences of words into flowing sequences of musical notes.

By looking at the overall behavioural trends, The Listening Machine can make measurements of the collective as a whole, which are displayed on the gauges described below. If the overall activity within the “sports” classification is high, its dial will indicate as such.

See the live data on the front page.


Fig 1. Sentiment

This triangle displays the group's collective mood, along three axes:

Positive
Negative
Neutral
This affects the mode and tonal style of the piece.


Fig 2. Rate

The current collective rate of status updates is indicated by this dial, from minimum to maximum. This has been calibrated based on the group’s historical activity.


Fig 3. Classification

This set of gauges indicates the current level of activity within individual sectors, based on how much a particular topic is being discussed.


Listener 1 Phonomat 2 Classification Sentiment Prosody Conductor 3 Encoder 4 The Internet
1 Listener

Listens for twitter status updates from a list of 500 people

2 Phonomat

Analyses each status update phonomat based on its subject classification, sentiment (positive/negative/objective) and prosody, or rhythm of speech.

3 Conductor

Generates the musical output, based on individual speech patterns and overall properties of the system (the rate of tweets, collective sentiment, etc)

4 Encoder

Broadcasts the live audio stream to the internet


From text to score

This diagram shows how a sequence of words can be transformed into musical notes by mapping their syllables to pitches of a scale. Consonants are dropped, leaving only the vowel sounds. These are ordered based on their typical fundamental frequency for an English speaker.

By preserving the rhythms and dynamics introduced by punctuation and stress, we can produce surprisingly structured-sounding motifs from simple sentences.

Prosody diagram