I chose a yellow accent and the numbers turned dark

That is on purpose. White on a pale accent cannot be read, so the Blocks and Ribbon designs switch their text to near-black, or to black on a mid-tone where only black reads, on any accent where white falls below the WCAG AA contrast of 4.5:1.

The designs that draw the numbers in the accent itself rather than on a tile of it — Plain, Outline, Pill, Panel, Stack and Stripe — do the same thing the other way round: a pale accent is walked towards black, a step at a time, until it reaches 4.77:1 against the background behind it, and on a dark Background it is walked towards white instead. Only far enough, so it is still recognisably your colour, and an accent that already reads is used exactly as you chose it.