Whale & Dolphin
Orchestra

Whale & Dolphin Orchestra.

Working prototype

Marine biodiversity has a sound of its own. This orchestra transforms real recordings from DCLDE and other waters into music for human ears, guided by measured rhythms and pitch contours. Choose your musicians on the map, or hear a finished piece.

Select recordings on the map and listen here.
01 / The ocean stage

Every voice has a place.

Preparing the programme…
Select a site to add a voice. Scroll or pinch to explore.
On the stage
    02 / Your ensemble

    The orchestra

    Arrangement

      Behind the listening experience

      From a voice
      to a musical score.

      Real recordings enter the system. Signal analysis measures their phrases. Those measurements become a playable guide, which a generative model uses to compose music.

      01 / Input · Recorded

      Listen to the animal.

      Short excerpts of hydrophone recordings: humpback song, dolphin whistles, sperm-whale codas, and killer-whale calls.

      Audio + recording location
      02 / Schema · Measured

      Measure the phrase.

      Onsets, durations, click groups, and pitch contours are extracted from audio. Tonal calls and clicks have different representations.

      Events + time + frequency
      03 / Guide · Constructed

      Give it an instrument.

      Contours become tones and clicks become mallet strikes. Each player's instrument and register are combined with your trims, timing, and arrangement.

      A playable combined guide
      04 / Output · Generated

      Compose a response.

      ACE-Step 1.5 transforms the combined guide into music. A deterministic response provides a comparison that follows the measured events by construction.

      Separate, playable audio stems

      The diagrams above illustrate the process. The spectrograms and phrase marks below come from the selected recording.

      Research foundation / DCLDE

      A diversity of calls,
      populations, and places.

      We selected excerpts from the DCLDE killer-whale detection and classification collection for its regional and acoustic diversity. Its authors describe it as the largest curated DCLDE audio-and-annotation dataset at the time of publication in 2025.

      Read the dataset paper
      1.6 TBaudio in the research collection225,000+annotations reported by its authors

      These figures describe the research collection. This prototype uses a small selection of excerpts, alongside Orcasound, OpenWhistle, and Dominica Sperm Whale Project recordings, to bring different waters and species into the same composition.

      Human and marine-mammal hearing differ. Registers can be shifted into a musical range, while measured event times and contour shapes remain the guide. The timing scores below test how closely the generated music follows the measured event times.

      NOAA: the variety of marine mammal sounds ↗
      Look inside a phrase

      The recording becomes the score.

      What is being tested?

      Does the music follow the phrase?

      Timing scores compare musical onsets with the animal's measured onsets, against the same output shifted in time. They help distinguish a response to this phrase from music that simply sounds appropriate.

      What is the artwork?

      A musical interpretation.

      The recordings are animal sounds. The measurements are signal analysis. The guide is a human-designed mapping and the composition is generated music. This process does not decode animal meaning.

      Development & models

      Tools, choices and authorship.

      We used Claude Code and OpenAI Codex to help develop and debug the software. Livia Zaharia and Anton Kulaga define the project concept, artistic direction and methodological choices, make the final decisions, and take responsibility for the work.

      Inspect the source code

      Measuring the recordings. Signal analysis extracts call timing and pitch contours. Our code maps those measurements to instruments and registers, then combines them into a playable audio guide.

      Generating the music. We run ACE-Step 1.5's pretrained base checkpoint locally. Its diffusion transformer generates music from the combined audio guide and a caption. Our current pipeline leaves the model's optional language-model planning stage disabled.

      Checking the response. A deterministic musical response provides a comparison, and timing scores test how closely the generated music follows the measured events. Original recordings, constructed guides and generated music remain separately playable.

      Generation settings and phrase tests
      Fine-tune playersTiming, trims, gain & register

        Add players in the concert hall to adjust their phrases here.

        Composition model
        Task

        In the first study, cover follows the guide's timing more closely. Lego tracks mostly ignore it.

        Engine logSession activity
        Nothing composed in this session yet.
        Physical installation

        An ocean you can
        gather around.

        Visitors gather around a printed ocean map, choosing recordings with buttons beside small whale and dolphin figures. A large wall screen shows how the calls become music.

        How a visit works
        AI-generated concept of visitors around a printed world map table with ivory whale figures and selection buttons, speakers around the room, and a large wall screen showing recorded, measured and generated sound
        The listening roomAI-generated concept
        Proposal · not yet built

        The website is the working prototype. We would develop the room with scientists through a residency or other funding.

        How a visit works

        1. 01 / Choose

          Choose your voices

          Each button adds one recording and plays a short preview of its original call. It lights while selected; press again to remove the voice. Up to six recordings can join, with the screen showing when the ensemble is full.

        2. 02 / Compose

          Hear the piece

          Press Compose on the shared touchscreen. The wall screen follows the recorded spectrogram, measured onsets and pitch contours, and generated music with one playhead. Each stage is labelled, and the musical guide is available to hear. Saved pieces play while a new one is made.

        3. 03 / Listen

          Listen again

          A bench with headphones offers a quieter place to compare the original calls with the music. A QR link lets visitors replay their saved piece at home and share it.

        AI-generated close-up concept of a hand pressing a selection button beside an ivory whale figure on a printed ocean map, with other animals on coloured bases
        The figures and buttonsAI-generated concept

        The table and figures

        Ivory 3D prints, about 10 to 15 cm long, would have sturdy fins and broad bases. The species are recognisable by shape; a coloured ring, number and site label identify each recording. Lifting a figure leaves the music unchanged.

        A matte ocean map on an oak table, roughly 2.2 × 1.2 m, stays readable in ordinary gallery light. Coastal insets separate crowded sites. The map marks recording locations; figures and buttons sit near the edges, within reach of seated visitors.

        Six speakers could give the original calls and their musical guides separate positions around the room. The generated composition plays as one shared mix. Table height, circulation and sound placement would be tested in the space.

        Practical detailsWhat we need to build

        Start with stereo

        Wired buttons connect to the app through a small controller. A computer runs playback and the large display, with stereo speakers or headphones. Saved pieces need no GPU; composing new pieces live needs the engine and a GPU computer.

        Develop the room

        A multichannel audio interface would route the calls and guides to the six speakers. We would synchronise the screen with playback and measure generation times during rehearsals.

        During a residencyWork with researchers

        Possible collaborations at HIFMB

        These are starting points for the ArtWaves residency. The researchers we work with would help decide which to pursue.

        Recordings and shipping noise

        With Marine Acoustic Underwater Diversity, we could add recordings, check measured phrases against expert annotations, and use a shipping-noise layer to explore how calls become masked.

        Listening and protection

        Marine Conservation and Marine Governance could inform map layers showing protected areas and shipping lanes. With the qualitative research lab, visitors' choices and comments could support a study of responses to ocean sound.

        Improving the prototype

        The prototype was built in three days with models that were ready to use. A residency would give us time to find a better fit for whale and dolphin recordings.

        Read the current method

        We could compare bioacoustic detectors with the current onset and pitch tracking, and try other music models alongside ACE-Step 1.5 or tune them on the guides.

        Musicians and scientists would help choose instruments, registers and timing. We would also extend the timing scores with researchers' measures to test how closely the music follows the calls.

        Sound to silver · an optional extension

        Sound to silver.

        The orchestra turns whale and dolphin calls into music. These studies follow the same kind of recordings into Livia's cast silver: a whistle that bends a real ring without breaking it, rings that keep moving after casting, a groove that stores a whistle, and the ideas still waiting for the workbench.

        Status

        Optional, studies so far. Nothing has been cast yet. These pieces would be made only if time and funding allow, alongside the orchestra and the listening room.

        01 / Bending a real ring · study run

        A whistle bends Inline and Roots by millimetres. The finger hole never moves.

        Two of Livia's stoneless rings, at full print resolution, are bent by recordings. Time runs along the metal from one end of the open ring to the other. Pitch lifts each cross-section along the finger, level swells the outer metal and pitch slope leans it.

        This builds on a sound-responsive technique Livia had already explored in her Inline Ring (2022). Its Grasshopper definition is in her “sound manipulation” folder: it takes microphone level with Sound Capture and uses it to move points along a curve. Here, that approach is applied to measured whale and dolphin calls, with casting checks to keep the resulting bends workable in silver.

        Every point keeps its angle around the finger and only moves outward or along it, so the ring cannot tear or pass through itself. Casting checks on the full print file then find the largest push that still casts: walls of at least 0.6 mm and openings of at least 0.4 mm, assumed lost-wax limits for the caster to confirm.

        Up to 4 mm
        Roots moves at its safe limit.
        About 2 mm
        Inline with lift alone, at exactly the same silver weight.
        0.000 mm
        Change in the finger hole across all 20 runs.
        2–28 %
        Surface made too thin by the usual audio-reactive push at the same strength.
        The bench, live. Choose a ring and a recording, then Play and bend: the ring bends as the sound plays. The push slider is marked at the verified limit, and Whistle live records from your microphone.Open full screen
        Inline ring bent by lift only, the original shown as a grey ghost
        Inline, lift only, SW_Nana at its 2.22 mm limit. The band follows the whistle's slow rise along the finger.
        Roots ring with all three gestures applied
        Roots, all three gestures at 3.70 mm. The cups lift and lean by up to 4 mm.
        Roots ring pushed along its surface normal, failing areas in red
        The usual push along the surface normal at the same strength: the shank erodes and the cups pass through themselves (red).

        Whistles carry the change; a sperm-whale coda, all clicks, only swells the band by about 0.5 mm. Report: Sound to silver: bending a real ring.

        Regenerate the pages with uv run --group art --group mesh main.py art silver and art silver-motion; the app shows those when present and a committed copy otherwise (bun scripts/silver.ts). Measurements come from signal analysis and casting checks; FLUX images are generated concepts.

        Back to the concert hall
        About

        An artist and
        a bioinformatician.

        Whale & Dolphin Orchestra is made by Livia Zaharia and Anton Kulaga. It began as a pre-project prototype for the HIFMB × HWK ArtWaves residency proposal, bringing marine recordings, signal analysis and generative music together with Livia's work in silver.

        Portrait of Livia Zaharia
        Architect · parametric jewellery artist

        Livia Zaharia

        Livia is a Romanian architect and parametric jewellery artist who works under the label Paral Design. She writes the geometry of her pieces with Grasshopper, COMPAS and Python, 3D-prints them, casts them mostly in silver and combines them with natural materials such as amber, walnut husks and quartz. Her work has been shown at Romanian Jewelry Week every year since 2021.

        In Livistone, her jewellery grows into buildings in a 3D town you can walk through in the browser. She founded GlucoseDAO, which builds open-source tools for glucose prediction, and contributes to the Longevity Genie ecosystem. Her rings are where the silver ideas for this project start.

        Portrait of Anton Kulaga
        Bioinformatician

        Anton Kulaga

        Anton is a bioinformatician at the Systems Biology of Aging Group, which develops computational tools to understand ageing. For this project he built the software: the map, the analysis of the recordings, the composition pipeline around ACE-Step 1.5 and this website.

        Open source

        The code and the method are public.

        antonkulaga/whales

        The repository holds this website, the Python engine that measures the recordings and composes the pieces, and the research behind both: datasets, models, art precedents and the silver studies.

        Recordings come from Orcasound, OpenWhistle, the Dominica Sperm Whale Project and DCLDE. Photographs of Livia's jewellery and her portrait: Livia Zaharia. To talk about the project, write to Livia on Instagram or LinkedIn, or open an issue on GitHub.

        See the installation plan