It’s quite simple really, the damage doubles after the first succesful hit, starting at 1 damage at the first hit. Meaning that at the 17th hit, you’d be doing 65.536 damage precisely. Damage is not reduced by Safety Augment or defence. As far as I know, this is the only normal Technick or attack for that matter that can break the damage limit (not counting Quickenings, or Esper Technicks).
There’s a catch however.
Numerology has an intial 85% chance to hit. Here’s a rundown:
First hit: 85%, 1 damage
Second hit: 85%, 2 damage
Third hit: 84%, 4 damage
Fourth hit: 83%, 8 damage
Fifth hit: 81%, 16 damage
Sixth hit: 79%, 32 damage
Seventh hit: 76%, 64 damage
Eighth hit: 73%, 128 damage
Ninth hit: 69%, 256 damage
Tenth hit: 65%, 512 damage
Eleventh hit: 60%, 1024 damage
Twelfth hit: 55%, 2048 damage
Thirteenth hit: 49%, 4096 damage
Fourteenth hit: 43%, 8192 damage
Fifteenth hit: 36%, 16384 damage
Sixteenth hit: 29% 32768 damage
Seventeeth hit: 21% 65536 damage
Eighteenth hit: always misses.
Hope this helps.
I think, although I’m not sure, that it resets whenever you move from one screen to the next. It will carry between enemies in the same area, as long as you don’t use another command apart from Numerology.
However, it usually misses at around hit 9 or 10 whenever I use it so it’s basically useless.