What is the Shape of Music? A Fractal Sound Analyzer

What is the Shape of Music?

The other tools on this site have tangible output; you can see it, touch it, or read it. Music as an output is untouchable; we can certainly interact with it, but the results are ephemeral. That means we have to think of another way to quantify the “shape” of music.

How can we do this? Sound is perceived by the ears, and then it vanishes. Luckily for us, notes can be measured and recorded as data. In this tool’s case, we are going to break the piece into tiny parts and measure the shape of each one.

In particular, we are going to give every tiny slice of music a set of coordinates in space, drawn from the way it sounds, and then watch the path those slices trace as the piece plays. Music that returns to a similar sound returns to a similar place; the path loops, clusters, and drifts, and that traced path is the “shape” we are after.

How Do We Analyze Sound?

To analyze a downloaded sample, we first prepare the music by cutting the file into 46-millisecond sections of time. Cutting it this way gives us a steady stream of equal-sized pieces we can turn into data.

Each piece is fed into a math operation called a Fourier transform, which takes a slice of sound and splits it into the individual frequencies inside it, along with how strong each one is.

We hear pitch unevenly: low frequencies we can tell apart very finely, but the higher we go, the more we lump neighboring pitches together. To match that, the frequencies are remapped onto the Mel scale, a scale spaced the way our ears actually resolve pitch rather than by raw Hertz.

We then take the logarithm of those Mel values, which mirrors the compressed way loudness feels to us, and run them through a second transform, a discrete cosine transform. From its output we keep the first thirteen numbers. Those thirteen, the Mel Frequency Cepstral Coefficients (MFCCs), are a compact fingerprint of the timbre of that 46-millisecond slice. Play a whole piece through this and you get a long stack of these fingerprints: thirteen numbers per slice, one slice after another.

Now the pretty part. Our tool takes this stack and finds the three directions in which the numbers vary the most, and makes those the three axes of the space.

Finally, the points are placed in the 3D view in time order and connected with a line, so you can follow the music’s path from start to finish.

Beauty as Sound

And what could be more beautiful than sounds crystallized, playing softly and spinning gently; a galaxy unearthed, the same as the tool above, only optimized to feature the harmony of shape. Magic.