Training My Model on a Bad Scorer Made It (Slightly) Better

The tl;dr first

In Part 1 I took apart the music reward that Xiaomi released with MiMo-V2.6. Tuned random notes beat every real piece I scored, including all of Bach. Among other reasons, I called it a bad scorer, but I was curious so I trained my own piano model against it anyway.

In short: The scorer made my model outputs slightly more “interesting” but there are still many issues, including, as I hypothesized in the previous article, seemingly incentivizing the model to avoid musical expression.

• The model: This is my own model, “Neurallegro” (to be renamed...), a small (25.5M-parameter) transformer I trained on musicXML data tokenized with my own music notation tokenizer (its own token format, not MusicXML, ABC or MIDI, so it can’t be fed to the scorer directly) I designed this past Spring
• The training: 300 steps of GRPO, a standard reinforcement-learning method for language models, using MiMo’s scorer as the reward. It ran for about 70 minutes on my laptop GPU.
• What happened to the score: on 928 held-out test pieces, the mean score rose from 0.606 to 0.772.
• What happened to the music: This is just an initial experiment to get an idea of what direction the training had gone in. I listened to 15 pairs blind, an untrained piece against a trained one from the same prompt, without knowing which was which. I preferred the trained one in 11 of 15. The main thing I heard wasn’t “better” so much as more interesting: more variety, and less repetition.
• It didn’t find the Part 1 hacks: None of the tricks from Part 1 (staccato spam, silent voices, note-value games and so on) became more common, and re-scoring every piece with those tricks removed leaves the gain intact.
• But the model also got less expressive, partly in the way I predicted in the previous post. The outputs had fewer articulations, dynamics, fermatas, etc (more on that later)

So, slightly better music, from a scorer I do still think is bad. This post is about why both can be true, and what changed in the music, both directionally “good” and “bad”. However, keep in mind this is just one run on one small pretrained model, and importantly, with just one listener (myself). This is in no way a rigorous scientific result, just a piece along the way as I work more on RL for music notation.

In Part 1 I said the scorer was a problem because it was on its way to actually be used in real training frameworks. Since Part 1 (as of October 14), neither pull request has merged (#3704, #843), and a second NeMo-Gym port (#3985) doesn’t include #3704’s caveat that the reward is only “a music-distribution proxy”. An automated reviewer on #843 had also independently flagged the same off-grid blind spot Part 1 found.

More details

Setup

Neurallegro reads and writes a tokenized form of MusicXML: notes, rests, voices, clefs, dynamics, hairpins, slurs, articulations, pedal and so on, as notated. It was trained on tens of thousands of piano scores. It isn’t open source (yet), so this post shows only a few of its outputs.

MiMo’s scorer expects ABC, so each output then takes this route:

1. Neurallegro writes tokens.
2. The tokens are decoded to MusicXML.
3. The MusicXML is converted to ABC (xml2abc).
4. MiMo’s released scorer runs unmodified: ABC → abc2midi → MIDI statistics → a score from 0 to 1.

Everything the scorer saw went through this route; the audio was generated from the MIDI files to MP3 using the Salamander Grand Piano soundfont (by Alexander Holm, CC BY 3.0) in FluidSynth.

Prompts. Neurallegro continues whatever it’s given. The test set had four kinds of prompt:

• Header only: a key signature, a time signature, clefs and usually a tempo, with no notes at all. Every note in the result is the model’s.
• Openings: the first two bars of a held-out piece from the PDMX corpus (pieces the model never saw in training).
• Seeds: short openings, 2–4 bars, that I wrote for testing, each in a different style (e.g. a generic march)
• Tunes: simple arrangements of a few well-known tunes (Ode to Joy, Twinkle Twinkle and so on).

The scorer scores the whole piece, prompt included. For openings, seeds, and tunes, a few bars of the scored music were therefore not written by the model. The header-only prompts are the cleanest test, though it is also interesting to look at how the score changes for the pieces with longer prompts.

Training. Each step, the model wrote 8 continuations for each of 8 training prompts (header-only prompts and openings from a separate validation split). The scorer scored all 64. GRPO then nudged the model toward the continuations that scored above their group’s average. That is 19,200 scored pieces over 300 steps. There was no penalty for drifting away from the original model.

What the scorer said

For comparison from the previous post using the same scorer:

Where did the gain come from? About +0.10 of the +0.166 came from more pieces getting past the converter’s checks at all (82% → 95%), and about +0.07 from the passing pieces scoring higher.

• Fewer rejections. At step 0, the single biggest cause of a zero was a tie pointing at a note that isn’t there, which abc2midi rejects. Pieces with that error went from 78 to 10. Another interesting point about rejections: I did a similar post-training run with a Qwen model (Qwen3-4B, writing ABC directly; more on that in the next article). In a test before that run, the starting Qwen model (before any training from me) only wrote music that got a score 14% of the time, even with formatting help in the prompt, whereas the base Neurallegro model did 82% of the time. It’s a different task (Neurallegro’s notation goes through a converter that handles the ABC formatting, which is where many of Qwen’s failures were), so it’s a rough comparison, but Neurallegro, specifically trained to write sheet music, does a better job writing music that converts through the abc2midi pipeline even before being trained with MiMo’s scorer.
• Higher scores among the passing pieces. Most of that improvement shows up in the scorer’s rhythm and register statistics. That’s the same direction Part 1 suspected the scorer rewards (more notes, shorter rhythms, a lower bass, a wider range), except here the model wrote busier music rather than putting staccato dots on the same notes. So it didn’t find the specific Part 1 tricks, but it did move toward what the scorer already favours.

By MiMo’s scorer, the trained model now beats the canon among pieces that got a score (0.810 vs 0.780), and by a wide margin over all pieces (0.772 vs 0.518). The wide margin is mostly because so many canon pieces are rejected for conversion problems (Part 1).

It also beats 2-voice random notes, but not 3-voice random notes. Part 1’s tuned noise (0.978) is still ahead of the trained model’s best piece (0.957, another O Come All Ye Faithful continuation, shown after the first comparison below). Training didn’t raise the single best score, either: the base model’s best piece scored 0.969 (a march continuation, shown after the third comparison). High scores did get much more common, though: 98 of the 928 trained pieces scored 0.9 or above, vs 28 before training.

Did it hack the scorer? I checked for several tricks from Part 1 that can be detected automatically:

• Note-value games: rewriting the same sound with different note values (checked by shifting the scorer’s measuring grid, the blind spot this trick exploits).
• Decoration spam: staccato, plus extra fermatas or trills.
• Silent voices: voices set to zero volume.
• Meter tricks: bar-length games (not possible here, since Neurallegro writes music bar by bar and xml2abc writes a bar line for every bar).

None of them became more common, at the final step or at any of the five checkpoints in between. Loops, Part 1’s biggest trick, actually became less common (see “Less copy-paste” below). At the very least, my model did not discover the known hacks from Part 1.

The listening method

I drew 15 test prompts at random, and for each one, one piece from before training and one from after, played in random order as “version A” and “version B”. The question was “Which of the two is the better piece of music?”, and I had to pick one. I only learned which was which after picking the best of each pair and writing my initial comments.

Results!

I preferred the trained piece in 11 of 15 pairs. What does “prefer” mean, though? In some cases, it was obvious. One version might have seemingly random chords with no melody, while the other might be a full piece with an A and B section (not to be confused with versions A and B), good melodies and harmonies, and reasonable cadences to end phrases (I tried not to penalize a piece if it was truncated when hitting the model’s output limit). Frequently, both versions started well but one version was very repetitive while the other explored a more “interesting” B section.

I was blind judging only by audio, so when you see the comparisons below, you will likely note that some of the versions I preferred turned out to have fairly messy sheet music, with incorrect voice membership or odd phrasing. This was intentional, however. The scorer also does not see the sheet music, only the MIDI made from it, which I think is a problem for teaching a model composition. Even as a musician, when I judge without the sheet music, I too am less able to discern what is playable and well-written for performers if the audio sounds reasonable.

As above, this is not a very rigorous test. I’m just one listener, I built the model so I already had an idea of what its outputs sounded like, and I only listened to 15 pairs (a split of 11–4 or more lopsided happens by pure chance 12% of the time, i.e. p = 0.12). I plan to do a much larger listening test soon. I was just curious about the direction of the training, and if there would actually be a difference or not.

I now give three example pairs to compare and contrast how the output changed after training. These are not a random sample of the model outputs, I chose these specifically to show a clear trained win (second example), a less clear choice (first example), and an example with lots of expression to look at (third example), plus it lined up with the best scoring outputs generated for the base model and the trained model.

Continuation of “O Come All Ye Faithful”

The first example is a continuation of the first measure (plus pickup) of “O Come All Ye Faithful”.

The base model version is very pretty and a reasonable continuation of the traditional tune. Its first phrase is 8 measures and its second phrase is 7 measures, but I do quite like the way it ends regardless. I think the B section is lovely, and to be honest I would love to learn this on piano as it does not look very difficult! (Although I would like to learn basically all the outputs)

The trained version does two 8-bar phrases and is also very pretty. When listening blindly, I picked this one because the melodies were more interesting and it had very notable countermelodies that the first version did not have. However, listening blindly I could not see the sheet music. Most of the sheet music is fine, though some of the voice crossing is a bit odd, and m14 has a chord with the note A3 occurring twice. Overall, I think the trained version is a slightly better piece of music with more interacting voices (polyphony), but its sheet music needs some work.

Additionally, one of the “O Come All Ye Faithful” continuations got the best score out of all of the final trained model’s outputs. Its sheet music has some issues (for example, m5 has a repeated note in a chord and some beaming problems), and I think some of the non-diatonic notes are a bit unexpectedly dissonant at times, but overall I quite like this version.

Header-only prompt in 3/4

Both of these pieces were prompted with simply the 3/4 time signature, 4-flat key signature and a tempo of quarter note = 96.

The base model version repeats the same fine but ultimately fairly boring (in my opinion) 8-bar phrase twice verbatim. As with the previous example, the trained version has more notation issues (seemingly unnecessary extra voices with rests; m38 has two tied 8th notes that should be combined into a quarter note, and the last measure is missing an 8th note, which is almost correct except that the piece started with a quarter note pickup so the last measure should be missing a full quarter note instead, if that is the convention it is trying to achieve). As with the previous example, I think the trained version has significantly more interesting countermelodies and interacting voices despite a few notation issues and some odd phrasing.

I noticed in this example and the previous that the trained version allows more overlapping {8th 8th} with {dotted 8th 16th} which is one of the reasons my ear picked up more noticeable countermelodies. These are more fugue-like (contrapuntal) whereas the base model versions are closer to a single melody with accompaniment.

Continuation of a march

For this example, the model was given the start of a march-style seed I wrote back in August. I show this example to illustrate the model’s use of expression such as dynamics, slurs, articulations, and even an arpeggio. The prompt itself has more expression than other prompts, so the model (base and trained) knows to continue expression when some is given. It seems less likely to introduce it on its own (though there was a slur in the first example whereas none existed in the prompt).

The base model version has some very interesting variations on the original theme, and though I wish it had brought back the original theme at the end (the highest scoring version does, see below), I think most of the choices are reasonable. This version uses lots of dynamic and articulation contrast, making this version particularly expressive compared to other outputs.

Similarly, the trained version uses more expression than the other outputs, though with less contrast than the base model version (no slurs, and the dynamics it wrote itself only go from p to mf). I also think the trained version is very repetitive; once it settles on the {dotted 8th 16th, quarter} pattern with a cadence every 4 measures, it does not relent. However, I really like that it introduced an arpeggio on its own; I thought that was a lovely ending to this piece.

The best scoring piece of the base model, before training with the MiMo scorer, was another really fantastic continuation of this march. In fact, it scored higher than the best scoring trained piece from above, which I find interesting. This version also used a repeat sign, and though the phrasing surrounding it is a bit odd (17 measures), I actually really like it. There is an unnecessary clef change (bass to bass) in m27, but I get what it was going for, because the previous section definitely could have had the left hand in treble clef. Additionally, as before, some of the voice membership is a bit messy, but overall, this is my favorite output on the page. So the scorer and I do agree on something at least, because this is the highest scoring output out of all samples generated throughout this experiment. The ending is just delightful!

What did training with MiMo’s scorer change about the music?

Comparing all 928 test pieces before and after training (every difference below has a 95% confidence interval that excludes zero unless stated otherwise):

Busier, but not always longer.

• About 42% more notes per bar, in about the same number of bars and seconds.
• Chords got thicker, and the range got wider by about five semitones.
• Rest-only bars nearly disappeared (4.1% → 0.2%).

Less copy-paste.

• Bars that exactly repeat an earlier bar went from 20.6% to 7.9%, and repeated four-bar chunks fell by 85%.
• In the second half of a piece, 94% of bars were new material, up from 84%. There were 25% more distinct rhythm patterns.
• This matches what I heard in listening. The variety is in rhythm and texture, though (busier rhythms and thicker chords, and in some pieces, like the first two examples, countermelodies) rather than harmony (overall pitch variety didn’t change, though bar-to-bar harmonic variety rose slightly)

Less expression.

• Expression markings fell by about a third per bar: accents −65%, fermatas −61%, tenuto −57%, hairpins −53%, dynamics −45% and slurs −34%.
• Staccato fell 17% (borderline), and pedal didn’t change significantly.
  • I first suspected the staccato hack from the last post, but taking the staccatos out of these pieces barely changes their scores, so that’s not it. Staccato was still used in as many pieces as before (24%), just less within each piece.
• The dynamics that remained got narrower and louder: pp marks went from 108 to 15.
  • The march (third example) shows an example of narrower dynamics: in the base model version, the dynamics used are p, mf, f, and ff while in the trained version, the model chooses p, mp, and mf.
• Most pieces still use some markings (66% → 59%), there are just fewer of them.

Endings got weaker.

• More pieces ran into the length limit (8% → 16%) instead of ending on their own. The scorer barely penalizes a piece that stops mid-phrase; after training, truncated pieces still averaged 0.75 (finished pieces: 0.78).
• Among pieces that did end, fewer end on a proper final barline.

Did the scorer start pushing expression out? Sure seems that way! In Part 1 I predicted that “a model trained against this reward would learn to avoid fermatas, dotted ties and other normal notation that happens to fail this one converter.” The results are consistent with that:

• broken ties nearly vanished (pieces with one: −95%), which is good, since those are mistakes. On the other hand:
• repeat signs declined by more than 50%, and D.C./D.S., segno, coda and Fine marks nearly disappeared (19 pieces → 2);
• fermatas fell 61% per bar. At step 0, pieces with fermatas were rejected more often (72% passed, against 84% without).

Conclusions

• This doesn’t necessarily prove the scorer will perform well OR badly. The results were mixed. The pieces got slightly better to my ear, but things the model used to do very well also got worse. This is a very mixed outcome.
• Tuned noise is still beating everything my model produced. Much of the gain seems to come from the scorer training my model to stop writing notation the converter rejects.

For some brainstorming for the future, a better scorer would reward the variety and keep the expression where appropriate. It would read the notation itself, so dynamics, hairpins and fermatas count as expression, instead of (for fermatas) just breaking the converter. It would not zero a piece over one converter error, and it might penalize stopping mid-phrase. It would be a more holistic approach for the notation and audio/MIDI combined, rather than focusing on one file format.

Code and data: github.com/leahchilders/mimo-grpo-part-2 (aggregate numbers, the listening protocol and results, the reward wrapper and the example pieces shown here; no model weights)