This has become a once a week exercise for fun.
I of course started counting from one to see where it might happen. Then I decided to use the magic of SQL to help me go through all the numbers.
I found the format model in the documention and moved on from there. I think my original query was a bit more elegant, but this works.
TO_CHAR(TO_DATE('01/01/4712 BC', 'MM/DD/YYYY BC') + (rownum - 1), 'JSP' ) num
CONNECT BY LEVEL < 1000
WHERE INSTR( num, 'A' ) > 0;
Pretty cool stuff.