Tuned Noise Beats Bach: Reward-Hacking the MiMo-V2.6 Music Scorer

Part 2 is up: Training My Model on a Bad Scorer Made It (Slightly) Better.

The tl;dr first

Briefly, Xiaomi’s MiMo-V2.6 release (September 22) included about a thousand “music composition” RL tasks and the reward function used to score them. In their tech report, Xiaomi used these tasks for training experiments on a small 9B model. That reward is now being ported into two widely used open RL frameworks, NVIDIA NeMo-Gym and Prime Intellect’s environment library (both pull requests still open as of October 1, 2026). If they merge, it may be one of the ready-made music environments people train against.

I ran real classical music through it, 1,443 keyboard pieces by canonical composers from the public-domain PDMX corpus, from Bach and Scarlatti through Chopin, Debussy and Joplin.

• A third of them scored exactly zero. In most cases this wasn’t the music’s fault: the file failed the ABC-to-MIDI step, often because of how it had been converted to ABC in the first place. (more on this later)
• The rest scored a median of 0.80 out of 1.

Then I let a simple search algorithm write music against the scorer.

• Random C-major notes, tuned by the search and sprinkled with staccato dots, reached 0.978. That beats every one of the 958 real pieces that got a score, including all 377 scored Bach pieces (Bach’s best: 0.94). (More on this example in the hacks section.)
• Short loops do almost as well: one 4-bar pattern played twice, 335 characters of text, reached 0.97.
• Bach’s Goldberg Variation XXIII scores 0.44 (one of his lowest; more on why below).

Now we will detail some of the scoring issues, the hacks, and what music the scorer actually did like. If you’re a musician, see if you can guess what classical-style music this scorer might favor. Anti-clickbait result: the answer did not surprise me!

What the scorer does

The task gives a model a detailed brief, for example:

Compose a piano solo in E major, 81 BPM andante, 4/4, about 64 bars and about three minutes, scored for solo grand piano in 2 voices. Texture: singing right-hand melody over wide left-hand arpeggios. Open with 4 bars establishing the accompaniment before the melody enters on an upbeat. The main section runs on a 8-bar diatonic loop of vi–IV–I–V, unchanged throughout. Thin the middle section to two voices for two descending phrases, then return with everything for the fullest passage. Dynamics start at mp, push to f in the middle and settle back to mp for a sustained tonic close. Aim for the feeling of a spring in the hills. Use no ornament symbols; write every note out. Output the complete piece as ABC notation.

The model answers in ABC notation, a compact text format for sheet music.

The scorer then:

1. converts the ABC to MIDI with a tool called abc2midi;
2. computes 18 statistics from the MIDI (note density, polyphony, pitch range, interval sizes, “key certainty”, self-similarity, and so on);
3. scores each statistic by how close it falls to the typical range of about 4,000 “human” pieces (full credit inside it, partial credit just outside);
4. combines those scores, weighted by category, with a smaller term (15%) for how closely the piece’s overall mix of pitches, intervals and note lengths matches the reference, into a score between 0 and 1.

Two things are missing from that list.

It never reads the brief. The key, meter, length, instrumentation and chord loop asked for are never checked; the reward function doesn’t even look at the prompt. Our best-scoring hack (hack 2, below) is 16 bars of C major in 4/4 for three voices. It gets the identical score answering every one of the thousand briefs:

• all 1,000 of them asked for a different length;
• 823 asked for a different number of voices;
• 254 asked for a different meter;
• only 73 asked for C major.

To be fair, Xiaomi doesn’t hide this. Their code describes the scorer as “MIDI human-likeness scoring,” and notes that the briefs’ requirements are “the material for grading instruction-following, which this scorer does not do.” But in the released recipe it’s the reward those thousand detailed briefs are scored with, and that’s the version being ported to other frameworks.

It never grades the notation. So I don’t think it’s quite accurate to say it scores the model’s notation. The model being trained writes ABC notation, but the scorer converts that to MIDI and compares its statistics with a reference set of other MIDI files.

MIDI doesn’t preserve most of what notation encodes. It records which keys go down and when they come up. It doesn’t keep enharmonic spelling (A♯ vs B♭), clefs, staff assignment, beaming, or intentional markings like articulation symbols and hairpins. Those are the kind of things that matter for writing a reasonable part for any instrument.

I’m not claiming there’s no value in scoring the MIDI conversion at all. But as the reward for what Xiaomi labels a “symbolic music composition” task, it’s misleading: the model writes notation, and much of that notation is discarded or made ambiguous before anything is judged. So this scorer trains a model to write notation that converts well into the kind of MIDI the scorer likes, quirks of the format and the converter included.

In short: two different scores can produce identical MIDI files. The model is asked to write the score, but it’s graded only on how the score converts to MIDI. And even then (see hack 4 below), it still gets fooled by how the notes are written.

The hacks

All of these were run through Xiaomi’s released scorer code, unmodified.

1. A ~300-character loop beats almost everything. One 4-bar, 3-voice pattern inside repeat signs, tuned by a hill-climber, scores 0.95–0.97. The entire file below is 335 characters. Even a hill-climbed single bar repeated once (about 150 characters) beats the median real piece.

X:1
T:c
M:4/4
L:1/8
Q:1/4=100
K:C
V:1
|: g2 .g2 A2 .A2 | C .C f4 .E .[EG] | .c' f4 .[DF]2 D | [Bd] .A2 .F4 .[df] :|
V:2
|: .C3 [AB] D4 | A D2 [GB] .F, A,2 [D,F,] | .[D,F,] .[D,F,] .C, D, D [CE] .[EG]2 | A2 D,3 .[GB] .[F,A,] .F, :|
V:3
|: .C,,3 .C, .D,,2 .[D,F,]2 | .C,, .E,, F,,3 E,3 | [E,G,] .D, B, A, .D,2 D,,2 | .A,2 A,3 D,,2 D,, :|

2. Random notes tuned to the scorer

• The scorer does catch crude degeneracy. One repeated note scores about 0.12–0.26, a scale over a bass line 0.34, and a 4-bar broken-chord loop 0.53.
• But “random, diatonic, at a plausible density” slips through. A hill-climber starting from random C-major notes reached 0.94.
• Along the way it discovered that stacking three copies of the same note into a “chord” nudges the score up a little further. No human would write that, and visual sheet music can’t even represent this (see the max 2 same notes per stem in measure 21 of Figure 2). This is just a quirk of representing sheet music flattened.

3. Staccato spam. Dotting random notes staccato is a free ~3 points (0.94 → 0.97). abc2midi renders a staccato as a shorter note, and that nudges the note-length statistics toward the reference band. The scorer can’t see a staccato mark, only a note that happened to be short. It isn’t the timing blind spot from the Bach section below: with that bug fixed, staccato helps just as much. It works through the rhythm statistics. Shorter notes shift the mix of note lengths toward what the reference set has, and the scorer’s rhythm score goes from about 89 to 100.

4. Same sound, different score. The scorer measures rhythm in quarter notes, whatever the time signature, and never takes the tempo into account. So the same music, written with different note values at a matching tempo, can score very differently.

• Beethoven’s Ecossaise WoO 23 scores 0.77 as written, in 2/4. Rewrite it in 2/2 with every note value doubled and the tempo marking adjusted to match, and it scores 0.65. In 2/8 with every value halved, it scores 0.48. The audio is identical in all three.
• Across 47 real pieces, the three versions’ scores differed by a median of 0.13, and by up to 0.29 without a single sound changing.

As a composer, I agree the choice of note values and time signature isn’t neutral. 4/4, 2/2, and 2/4 read differently to a performer. But the scorer isn’t responding to any of that. Re-barring the same notes in a different meter (3/4, 6/8, 5/4…) barely moves the score. It’s responding to its own measuring grid. Shorter note values push more notes below the scorer’s eighth-note grid, which compounds with the blind spot in the next section.

5. Silent voices count. Adding a bass drone with its MIDI volume set to zero, roughly doubles (about 0.31 → 0.61) a melody’s score. That’s identical to an audible drone, because the scorer never reads the MIDI volume setting.

6. One word switches off the bar-length check. M:none (“free meter”) is valid ABC. It turns off the scorer’s bar-length check completely, so pieces with every bar deliberately overfull go from 0 to a real score.

7. The instrument doesn’t matter. Changing the MIDI instruments between piano, strings, brass and synth drums leaves the score exactly the same, even when the brief asked for a specific instrument, like a piano solo.

8. One blank line = zero. The scorer rejects any piece with an empty line inside the music. In ABC, a blank line marks the end of a tune, so there’s a reason for the rule, but a good piece with one stray empty line scores 0. Carefully converted files won’t hit this, but it’s an easy formatting slip for a language model: an empty line between the header and the music, say, or between voices.

The scorer can’t see Bach’s faster counterpoint

Bach’s median score (0.79) is about average. Among his major works, the dramatic lows are Goldberg Variation XXIII (0.44), the F-major Prelude from The Well-Tempered Clavier, Book I (0.46), and the E-minor Fugue from the same book (0.52). These low scores mostly come from the same timing bug, not from the music itself.

The converter abc2midi starts every note one MIDI tick (about a millisecond) late. The scorer checks “what’s sounding” only at exact eighth-note grid points. So any note shorter than an eighth never registers, and an eighth note registers only if it’s syncopated (starts between grid points).

In other words, the scorer can’t see polyphony or harmony at a fine enough resolution. Notes shorter than an eighth, and eighth notes that aren’t syncopated, never register in its polyphony, roughness, harmonicity or tension statistics. Only 6 of Goldberg XXIII’s 1,474 notes are visible to them, and across real music, a median of 57% of notes are invisible. The score jumps with this fixed: Goldberg XXIII 0.44 → 0.73, the Prelude 0.46 → 0.68, the Fugue 0.52 → 0.72. The hill-climber found this blind spot too. With it fixed, every hill-climbed hack drops 4–14 points. But fixing it doesn’t close the hole: re-running the hill-climber against the fixed scorer climbs right back to 0.93–0.98, and the best of those still beats every real piece scored the same way.

Counterpoint is harmony in motion, but to this scorer, fast counterpoint, some of the most studied harmonic writing in Western music, reads as having no harmony at all.

Not all of Bach scores low, though. His fugues that pass score a median of 0.82, about average; the lows are specific pieces built on running sixteenths. The two- and three-part Inventions and Sinfonias are lower (median 0.74). The chorales mostly don’t get a score at all, for a different reason: see fermatas, below.

Real music vs the abc2midi converter

Before any statistics, the scorer converts the ABC notation to MIDI. If abc2midi reports any error, or 10 or more bar-length warnings, the score is 0. A warning means a bar whose notes don’t add up to the time signature (five beats’ worth of notes in a 4/4 bar, for example), and each voice, and each pass through a repeat, counts separately. 34% of the 1,443 pieces scored zero this way. The pieces that failed would actually have scored higher (median 0.85) than the ones that passed, so the converter is zeroing music the scorer would otherwise have rated highly, often over a conversion problem rather than anything in the music.

Many of these errors come from the earlier automatic MusicXML-to-ABC conversion (ties that ABC can’t express are the most common), but some are genuinely valid notation that abc2midi rejects, a big problem when one error zeroes the whole piece. For example:

• Fermatas. By default, abc2midi plays a fermata at double length, so every bar with a fermata overflows, and each voice counts as a separate warning. A four-voice chorale reaches the limit after just a few fermatas. 85% of the 409 Bach chorales in the music21 corpus score zero, mostly because of this: about two-thirds of those pass when their fermatas are removed.
• A trill written on a chord. Valid ABC, rejected; it zeroes, among others, Chopin’s Étude Op. 25 No. 3, which scores 0.76 once that one trill is removed.
• A dotted tie, or a staccato on a tied note. Both valid in the ABC standard, both rejected.

Since the model writes ABC but the scorer judges the MIDI, the converter’s quirks are part of the reward, and not just for obscure notation. Fermatas are one of the earliest symbols taught to performers and composers, and an important part of musical expression. Plenty of famous pieces score 0.0, and some of them for perfectly standard notation: Fauré’s Dolly “Berceuse” for its fermatas, Chopin’s “Horseman” Étude (Op. 25 No. 3) for a single trill on a chord. And the longer the piece, the more chances it has to fail on conversion. Split the pieces into four equal groups by number of notes: 14% of the shortest group score zero, then 21%, then 39%, and 60% of the longest group. So most big sonatas and ballades never get a score at all.

The model might never run abc2midi itself, a model trained against this reward would learn to avoid fermatas, dotted ties and other normal notation that happens to fail this one converter.

So what kind of music does the scorer actually like?

From the 1,443 pieces I put through it, the answer is ragtime! Joplin’s pieces, mostly rags, score a median of 0.91, the highest of any composer with enough pieces to average across, and the top real piece in the whole set is The Entertainer (0.976).

This is mostly unsurprising. The scorer’s “human” reference set seems to be multitrack band or performance MIDI: about six channels, bass down to F1, busy and repetitive. Rags have the texture of popular band music. A rag is basically a one-person band: the bass player (a low note on the beat), rhythm guitar (the off-beat chord) and the lead (the syncopated tune).

Our hypotheses, from comparing the rags’ statistics with everything else:

• Dense, busy texture. Rags have a median of about 8.5 notes per beat, against about 3.9 for the rest, yielding good density and short-gap statistics.
• The left hand. A low bass note on the beat and a chord on the off-beat reaches the deep bass the reference set expects; rags go down to about G♯1, against about D2 for the rest.
• Wide range and clear tonality. A five-octave span, and a very clear home key: the scorer’s “key certainty” is about 0.95 for rags, against about 0.88 for the rest.

Rags lose almost nothing on rhythm, register, or density. Their total loss averages 9 points out of 100, against 23 for everything else.

A few other interesting notes

Random diatonic notes with no optimization at all score about 0.70 in two voices. Add a third voice and they score about 0.82, already level with the median real piece (0.80). The NeMo-Gym pull request itself calls the reward “a music-distribution proxy, not a prompt-compliance or listening-quality oracle.” But if it becomes a ready-made music environment in two major RL frameworks, people will train against it, which is why I wanted to take a look at it in the first place.

What a sound music reward would check

• The brief: key, meter, length, instrumentation, voices, the chord loop, the form plan. If a prompt is included, adherence to it should matter.
• The notation, not a (or, in addition to) MIDI render: spelling, voices, ties, articulation, etc. If the goal is ever to write performable music, the notation matters.
• Time measured properly, so short notes count and rewriting the same sound doesn’t change the score.
• Instrument-appropriate references: piano judged for piano, string quartet judged for string quartet, not everything against one band-MIDI-like reference set.
• Actual music theory where the brief asks for it: voice leading, cadences, resolution.

Code and data: github.com/leahchilders/mimo-reward-hacking-part-1