I wouldn't worry too much about compatibility with all Famicom and NES games. Isn't this supposed to be a new console that only uses NES compatibility as a fall-back for programmers that don't want to dedicate too much time to a system no one has?
BTW I started drawing up my own fantasy system some time ago, also using the 65816:
Numbers and limitations are arbitrary and don't have much to do with reality.Fantasy Computer Fancom
A fantasy 8-/16-bit computer and video game console.
CPU: 65816
Video: ELF-VPU
Sound: SIREN-APU, Z80 core
ROM: BIOS, DWARF-BASIC (changeable)
RAM: 16 kB (expandable)
Save Memory: 64 kB EEPROM (expandable)
Ports: 4 controller ports, PS/2, USB, Cassette Tape, Expansion port, Cartridge port,
SD-card port, RGB video, AV (RCA), S-video, 3.5 mm stereo headphone, HDMI.
Video Processing Unit
ELF-VPU, a custom chip that is similar to the Yamaha VDPs for MSX,
Hudon's VDC for PC Engine and Nintendo's PPU.
VRAM: 64 kB
Colors: 24-bit RGB, 16x16 palettes (shared by sprites and BG characters)
Sprites: 64 sprites, 16 on a scanline, hardware collision, sizes: 8x8;
8x16; 16x16 (mix freely), zooming, x-/y-flipping.
Background: 2 BG layers and a backdrop color, 8x8 characters, 256x224 dot.
Scroll: Both backgrounds, both axes, pixel-by-pixel.
Display Modes: Text, Graphic, Bitmap.
Interrupt: Vblank, collision and sprite overflow.
Output: RGB, S-video, composite (NTSC and PAL), HDMI.
VRAM content:
Background Name Table (BGNT)
Sprite Attribute Table (SPAT)
Pattern Table (PATT) (shared by sprites and BG characters)
Palette Table (PALT)
Audio Processing Unit
Channels:
Rectangle
Triangle
Sawtooth
Noise
Programmable Waveform
PCM
Databus is 16-bit. Address space is 24-bit.
Save Memory is used to store system settings, but can
also be used by other games and programs for saving data in a
non-volatile memory.
There is also a points system, where points are saved in a region of
the Save Memory (like Play Coins of 3DS).
Besides BASIC, a text editor, built-in games, a settings menu and a
65816 assembler are also included.