CMP setting N flag when it shouldn't?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

Ringeru
Posts: 2
Joined: Sat May 04, 2019 12:36 pm

CMP setting N flag when it shouldn't?

Post by Ringeru »

Hi everyone,

This is probably a newbie question, but I don't fully understand the behavior of CMP in 6502 assembly. When comparing two numbers (say accumulator with something in memory), I thought it set the N flag when the accumulator is smaller than memory. However, in the following example, it sets N even if A > #$22

Code: Select all

LDA #$AA
CMP #$22
Can anyone explain what is happening here?

Thanks!

(Edited to correct a typo)
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: CMP setting N flag when it shouldn't?

Post by Pokun »

The N flag is the negative flag. It is set if bit 7 of result is set, and cleared if bit 7 of result is clear (so always a copy of bit 7 of result). Bit 7 is used as the sign bit in signed numbers, but the CPU doesn't really care if you treat a number as signed or unsigned, it will simply always set the flag the same as bit 7 of result.

CMP is actually a subtraction that doesn't affect the accumulator, it affects the N, Z, and C flags the same way as SBC a SEC, SBC-sequence does.

To check if A is smaller than the memory after a CMP, you check the C flag, not the N flag:
C=0: A < M
C=1: A >= M
This only works for unsigned numbers though.

You use the Z flag to check for equality (works with both unsigned and signed numbers):
Z=0: A != M
Z=1: A == M
So with a combination of C and Z flags you can do all types of comparisons of unsigned numbers with CMP (and CPX and CPY).
Last edited by Pokun on Sun May 05, 2019 4:06 am, edited 4 times in total.
Ringeru
Posts: 2
Joined: Sat May 04, 2019 12:36 pm

Re: CMP setting N flag when it shouldn't?

Post by Ringeru »

Thanks for the explanation, that makes sense!
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: CMP setting N flag when it shouldn't?

Post by tokumaru »

Ringeru wrote:However, in the following example, it sets N even if A > #$22

Code: Select all

LDA #$AA
CMP #$22
In this case, the accumulator is so large that it becomes negative. Since you're using the N flag, that means you're dealing with signed numbers. 8-bit signed numbers are in the range -128 to 127, so the $AA you have there is actually representing the number -86, not 170, which can't be represented as a signed 8-bit number. Since -86 IS less than $22 (34), the result is correct.

Keep in mind that there's no difference at all between signed and unsigned numbers as far as the 6502 is concerned. The bit representation is the same for both, and all results are correct for both, what changes is how the programmer interprets all the bits and status. Some flags are meant for signed operations, others for unsigned operations, and it's your job to respect the valid numerical ranges that are supported for each type and to interpret the results according to these definitions as well.

If you really need to compare signed values as large as 170, you have to bump your numbers to 16-bit, and bump the comparison to a 16-bit subtraction:

Code: Select all

lda #$AA ;low byte of $00AA
cmp #$22 ;low byte of $0022
lda #$00 ;high byte of $00AA
sbc #$00 ;high byte of $0022
Note that while CMP can be used to compare the lower 8 bits, SBC is needed for the upper 8 bits because it takes the carry from the previous operation into comparison, while CMP doesn't. With this you can safely compare numbers in the range -32768 to 32767.

If you don't need to work with signed numbers at all, don't use the N flag, use the carry flag instead, and you can compare numbers between 0 and 255 using just the one CMP instruction (no need to bump the math to 16 bits).
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: CMP setting N flag when it shouldn't?

Post by Pokun »

According to this, the N flag is "NOT the signed comparison result".
tokumaru wrote: Note that while CMP can be used to compare the lower 8 bits, SBC is needed for the upper 8 bits because it takes the carry from the previous operation into comparison, while CMP doesn't.
Ah yes, I said "CMP is like SBC that does not affect A". It should really be "CMP is like a sequence of SEC, SBC that does not affect A". I corrected my post above.


BTW Ringeru, one thing to remember is that the name of the flags reflects their most common usage, but not their only usage. The carry flag for example is used for arithmetic carry or borrow but also used for other totally unrelated things, and the N, Z and V flags can also be used for various not so obvious things.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: CMP setting N flag when it shouldn't?

Post by tokumaru »

Pokun wrote:According to this, the N flag is "NOT the signed comparison result".
I never realized this (haven't done much signed math), but that's correct. In the example they gave, the math is 127 - (-128), which is effectively 127 + 128, which causes an overflow because a signed 8-bit number can only go up to 127. So yeah, I guess that comparing against a negative number can easily cause an overflow, rendering the N flag useless. Hopefully the V flag will signal if this is the case (yeah, looking at the section about signed comparisons it says that you need both the N and V flags to find the result of a signed comparison).
one thing to remember is that the name of the flags reflects their most common usage, but not their only usage.
This is true for many other things in assembly as well, not just the flags. There are instructions like JSR and RTS which are normally used for calling and returning from subroutines, as their names imply, but when combined with a bit of stack manipulation they can be used for other purposes. There's even the simple case of BEQ/BNE, which mean "branch if equal/not equal" but you don't have to use it only after comparisons, seeing as comparisons are only one of the things that affect the Z flag, which's what ultimately controls the behavior of those instructions. Instructions, flags, addressing modes, etc. in assembly can often be used in more ways than their names imply.
niconii
Posts: 219
Joined: Sun Mar 27, 2016 7:56 pm

Re: CMP setting N flag when it shouldn't?

Post by niconii »

There's a similar edge case in how arithmetic shift right on a signed number is not equivalent to dividing that signed number by a power of 2. (-1 / 2) == 0, but (-1 >> 1) == -1.

EDIT: See below.
Last edited by niconii on Sun May 05, 2019 1:47 pm, edited 1 time in total.
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: CMP setting N flag when it shouldn't?

Post by Pokun »

Good to know! Is there any other cases where shifting doesn't equal multiplication/division with power of 2?
tokumaru wrote:I guess that comparing against a negative number can easily cause an overflow, rendering the N flag useless. Hopefully the V flag will signal if this is the case (yeah, looking at the section about signed comparisons it says that you need both the N and V flags to find the result of a signed comparison).
Yes the signed comparison result is in N XOR V after the subtraction. That's another difference between CMP and SBC I failed to mention, CMP does not affect V unlike a SEC, SBC-sequence (also CMP subtractions are not affected by the D flag, but that's irrelevant on NES). I've corrected my post again. So for that reason CMP cannot be used in signed comparison, SEC, SBC-sequence are used instead so that V is affected.

In order to do a signed comparison you can use a SEC, SBC-sequence and then use a trick to get the signed comparison result (N XOR V) into N:

Code: Select all

;8-bit signed comparison
  SEC
  SBC NUM    ;subtract NUM from A to compare them
  BVC label1 ;if V = 0 then V XOR N = N
  EOR #$80   ;1 XOR N, V XOR N = N
label1:
  BMI label2 ;if N = 0, A >= NUM, goto label2
  BPL label3 ;if N = 1, A < NUM, goto label3
label2:
label3:
Details are explained in the above-linked tutorial. Basically if V is cleared after the subtraction, then N already is the same as V XOR N. Else if V is set, EOR with $80 (N is bit 7) to get N = 1 XOR N = V XOR N. Now when result is in N, BMI or BPL can be used to branch.

Code: Select all

;16-bit signed comparison
  LDA NUM1_L
  CMP NUM2_L  ;compare low byte using CMP
  LDA NUM1_H
  SBC NUM2_H  ;compare high byte using SBC to include C and V
  BVC label   ;if V = 0 then V XOR N = N
  EOR #$80    ;1 XOR N, V XOR N = N
label:
  BMI label2 ;if N = 0, NUM1 >= NUM2, goto label2
  BPL label3 ;if N = 1, NUM1 < NUM2, goto label3
label2:
label3:
Higher than 8-bit is done the same way only each byte must be compared and the C flag must be included. Only the low byte can use CMP, the rest all have to use SBC (without a SEC) for the subtraction so the carry is included and so that the overflow flag is affected.

Signed comparison is useful if you are making an action game with acceleration-based movement. That way you can use positive and negative acceleration and velocity to move objects with.
Last edited by Pokun on Mon May 06, 2019 3:46 pm, edited 1 time in total.
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Re: CMP setting N flag when it shouldn't?

Post by Bregalad »

Nicole wrote:There's a similar edge case in how arithmetic shift right on a signed number is not equivalent to dividing that signed number by a power of 2. (-1 / 2) == 0, but (-1 >> 1) == -1.
Actually it does, but similarly to when divinding positive numbers by 2, the result is always rounded down. -1/2 = -0.5, rouded down it makes -1 so the result is correct.

If you want the result to be rounded up, you need to do an ADC #$00 after the shift (this works for both signed and unsigned numbers). (edit: I actually use this in my NES music engine to handle octave shifts of frequencies - without this the pitch tends to sound wrong !).

When it comes to the logic of V and N for signed numbers I've never fully understood it despite years of 6502 coding, but Tokumaru explained it greatly. Basically if V=1 the result stops being meaningful for signed numbers, and when V=0 then N is the sign of the result. But what happens when adding an unsigned 8-bit with a signed 8-bit, a situation that is less really rare in a game for example when moving objects and the coordinates are always positive but the speed can be negative ? Or when mapping a metasprite where the coordinates are always positive but relative position to hotpoint can be negative.
Last edited by Bregalad on Sun May 05, 2019 11:05 pm, edited 2 times in total.
User avatar
rainwarrior
Posts: 8735
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: CMP setting N flag when it shouldn't?

Post by rainwarrior »

Nicole wrote:There's a similar edge case in how arithmetic shift right on a signed number is not equivalent to dividing that signed number by a power of 2. (-1 / 2) == 0, but (-1 >> 1) == -1.
Another way to put this is that an arithmetic right shift on a negative number rounds down rather than toward zero. (Though C and C++ implement round toward zero with their division operator, some languages like Python round down in this way, and there are arguments for doing it that way, in particular how the modulo operator corresponds, but the different standards make it a point of confusion.)

This can be fixed with an increment before the right shift.
(-1 + 1) >> 1 = 0

For a signed arithmetic shift right you can detect sign and correct the rounding with an ADC #0. A pseudo operation for signed divide by two might look like:

Code: Select all

CMP #$80 ; move sign into carry
ADC #0 ; +1 if signed
CMP #$80 ; load the new carry
ROR ; right shift
If you need to do more than one shift, you need to reload the carry each time. The rounding up, on the other hand, can be done in one step (i.e. if >> 3 you can add +7 rather than incrementing before each shift). Code to divide a signed number by a larger power of two will probably want to branch on the sign bit and have different code for the negative and positive sides.


Many signed operations on the 6502 are a bit more complex than their unsigned counterparts in similar ways.


Edit: bregalad got to it while I was writing this, heh. I guess this is not entirely redundant though.
Last edited by rainwarrior on Sun May 05, 2019 2:21 pm, edited 7 times in total.
niconii
Posts: 219
Joined: Sun Mar 27, 2016 7:56 pm

Re: CMP setting N flag when it shouldn't?

Post by niconii »

Ah, yeah, you're right. Same goes for stuff like -3, -5, etc. so this isn't really an "edge case" at all.
supercat
Posts: 161
Joined: Thu Apr 18, 2019 9:13 am

Re: CMP setting N flag when it shouldn't?

Post by supercat »

rainwarrior wrote:
Nicole wrote:There's a similar edge case in how arithmetic shift right on a signed number is For a signed arithmetic shift right you can detect sign and correct the rounding with an ADC #0. A pseudo operation for signed divide by two might look like:

Code: Select all

CMP #$80 ; move sign into carry
ADC #0 ; +1 if signed
CMP #$80 ; load the new carry
ROR ; right shift
If you need to do more than one shift, you need to reload the carry each time.
Or else handle positive and negative values separately.

Code: Select all

    ; Get value/4, rounded toward zero into accumulator
    lda value
    bpl positive
negative:
    lsr
    lsr
    adc #$C0
    bvc done ; Adding negative value to positive number won't overflow
positive:
    lsr
    lsr
done:
To use floored division rather than truncating, one could substitute "ora #$C0 / bmi done" for the negative case.
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: CMP setting N flag when it shouldn't?

Post by Pokun »

Bregalad wrote: When it comes to the logic of V and N for signed numbers I've never fully understood it despite years of 6502 coding, but Tokumaru explained it greatly. Basically if V=1 the result stops being meaningful for signed numbers, and when V=0 then N is the sign of the result.
Yes if there was an overflow, a negative sign doesn't mean that the minuend of the subtraction is smaller anymore, but actually the reverse:

Code: Select all

V XOR N
0 XOR 0 = 0: no overflow, positive difference, minuend is bigger or equal
0 XOR 1 = 1: no overflow, negative difference, minuend is smaller
1 XOR 0 = 1: overflow, positive difference, minuend is smaller
1 XOR 1 = 0: overflow, negative difference, minuend is bigger or equal
So I guess basically if there was no overflow, positive means minuend can't be smaller than the subtrahend, and negative means it must be smaller. If there was an overflow however the reverse is true.

Bregalad wrote: But what happens when adding an unsigned 8-bit with a signed 8-bit, a situation that is less really rare in a game for example when moving objects and the coordinates are always positive but the speed can be negative ? Or when mapping a metasprite where the coordinates are always positive but relative position to hotpoint can be negative.
If mixing unsigned and signed numbers is a problem, I guess you may convert both 8-bit numbers to signed 16-bit numbers (so that the unsigned number fits) first. Then you can do 16-bit signed comparisons. I'm not sure adding is a problem though. I'm adding an object's velocity value to its position value each frame, and if the velocity is negative it will simply work like a subtraction and the object will move backwards.
User avatar
dougeff
Posts: 3079
Joined: Fri May 08, 2015 7:17 pm

Re: CMP setting N flag when it shouldn't?

Post by dougeff »

My opinion is that bit shifting only makes sense for unsigned numbers.

When you work with an 8 bit CPU... bit shifting is more a means to calculate PPU addresses, or like the example above, music code, where the data you're working with only makes sense as a positive unsigned value.

And 1 shifted to 1/2 should shift to 0. You're past the granularity of 1 pixel, just drop it to zero.
nesdoug.com -- blog/tutorial on programming for the NES
supercat
Posts: 161
Joined: Thu Apr 18, 2019 9:13 am

Re: CMP setting N flag when it shouldn't?

Post by supercat »

dougeff wrote:My opinion is that bit shifting only makes sense for unsigned numbers.
Bit shifting signed numbers left is equivalent to multiplication in cases where the arithmetical value of the product would fit in the result type. Bit shifting of signed numbers right is equivalent to floored division. In what way do those not make sense?
Post Reply