FMV on NES (dream I had)

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

User avatar
Marscaleb
Posts: 240
Joined: Fri Sep 11, 2015 10:39 am
Contact:

FMV on NES (dream I had)

Post by Marscaleb »

Okay, so this idea literally came to me in a dream last night.
I don't think anyone would ever bother doing this, but I am just wondering if this is technically possible.
First off, you'd need to create a custom cart. Now install a modern flash drive of some sort onto this cart so you'd have megs or even gigs worth of storage space. But the cart would interact with the NES like it was using... I can't remember, but I think it's CHR-ROM? The one that Mario 3 and Smash TV use so they can swap out tiles in the middle of drawing a frame.

With this set up, you play a literal full-motion video on a NES game. It would look kinda weird because it would be a four-color video, or possibly with some additional palettes set up within a grid, but still.

I imagine I'm not the first to think of it, but whatever. When I was first waking up this morning (and was still half asleep) it sounded really cool. As I've woken up it sounded less cool, but I'm still wondering how technically possible it is.
I'm sure it would be technically possible for at least a really short animation; someone could even create a ROM that plays in an emulator. The issue that I'm not completely certain on is when a video requires more pages of tiles than we've seen on an NES. If you wanted to, say, play a full-length movie. I understand that when the NES swaps out the tiles in its memory in that fashion, it's really just pointing to a location in its ROM, not actually loading data into it's memory. So would we hit a limit where the NES could no longer point to that many different locations in the ROM? Would this limit be easily bypassed with a new chip designed to handle millions of locations, would would we need to bypass the limit by over-writing the tile data in the "ROM," which would require more modern hardware to upload new tiles to the "ROM" fast enough to replace a whole screen during V-Blank?
...Or am I just wrong about this already?

At any rate, that was really just the first step. The next inevitable step that came to me while I was dreaming was: what if you made a cart that had its own processor on it, maybe something like a raspberry pi, and you actually tried to render out a fully 3D scene in real time, converted that to a video, and played it through the NES using the FMV system described above?
Obviously, there is no reason to ever actually do this. But you know, I'm curious and it's fun to think about.
I see two real issues with this. One, could we actually get the NES to have its tile data completely re-written fast enough to make this possible, which is just the FMV issue above. Second, could we actually get enough power to a NES cart to run a raspberry pi or some other co-processor capable of the needed 3D calculations? I'm sure you could create a cart with an extra power cable hanging out the top that needs to be plugged in. Or even install a cell-phone battery on the thing to power the new hardware. (You just gotta take the cart out of the NES and charge it every night so you can play it again tomorrow.) But how reasonable is it to get enough power straight from the NES itself to run this process?

I imagine you'd be looking at a game with incredibly simple 3D, mostly relying on wireframes, looking something like Spectre or Carrier Command. If nothing else, this is just to ensure readability, since you're only going to have four colors on the display. Sure, if you tried this same thing with the SNES you could have an image that was 256 colors in 15-bit color depth, and with that much detail you could render a modern PS4 title and still have it look pretty good. (Not to mention more buttons on the controller.) But the NES, man, that's where this becomes interesting.

But yeah, how possible is this?
lancuster
Posts: 235
Joined: Thu Feb 18, 2016 3:20 am
Contact:

Re: FMV on NES (dream I had)

Post by lancuster »

If you want to convert a sequence of images into an animation on NES ROM, you can use this utility:
https://kasumi.itch.io/ichr
If you want a full-screen video, this is unlikely to be done. The fact is that NES is not a Sega CD, a full-screen video can not be fits in. Unless you are good at programming, and you know how mappers works, and you are able to work with a huge number of compressed images (i.e RLE compression).
The only example of a full-screen video is a Bad Apple, but this is a self-written project, plus - the entire graphics are monochrome and have low detail.
Fiskbit
Posts: 891
Joined: Sat Nov 18, 2017 9:15 pm

Re: FMV on NES (dream I had)

Post by Fiskbit »

Cartridges can simply stream unique slivers (8x1 pixel strips and associated attributes) to the PPU. Examples of this include Doom on NES and Tom7's NES reverse-emulation work.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: FMV on NES (dream I had)

Post by rainwarrior »

There was this thing that a Netflix employee did as well, but it's not a very interesting demo.
User avatar
Marscaleb
Posts: 240
Joined: Fri Sep 11, 2015 10:39 am
Contact:

Re: FMV on NES (dream I had)

Post by Marscaleb »

Fiskbit wrote: Sat Feb 01, 2020 10:50 am Examples of this include Doom on NES
Holy crap that's amazing!
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: FMV on NES (dream I had)

Post by tokumaru »

There have been several ideas regarding FMV on the NES throughout the years.
Marscaleb wrote: Sat Feb 01, 2020 10:26 amI'm sure it would be technically possible for at least a really short animation; someone could even create a ROM that plays in an emulator.
Really short animations can be done using period hardware. The MMC3, one of the most common mappers from back in the day, can have a total of 16,384 tiles, which can be used to draw 17 unique 960-tile screens. You can squeeze a few extra frames if you make the video smaller than the full screen, but it'll still be just a short animation.

The MMC5, a more complex and not so common mapper, can manage a total of 65,536 tiles, or 68 full screen frames. Still just a few seconds of video at any reasonable frame rate.
I understand that when the NES swaps out the tiles in its memory in that fashion, it's really just pointing to a location in its ROM, not actually loading data into it's memory. So would we hit a limit where the NES could no longer point to that many different locations in the ROM?
The amount of memory a mapper can address depends on how many bits it dedicates to indexing memory banks. The maximum amount possible back in the day was 65,536 tiles, with the MMC5.
Would this limit be easily bypassed with a new chip designed to handle millions of locations, would would we need to bypass the limit by over-writing the tile data in the "ROM," which would require more modern hardware to upload new tiles to the "ROM" fast enough to replace a whole screen during V-Blank?
You could use larger ROM chips today and create new mappers to address more tiles, but they don't really make the kind of ROM chips the NES can directly access in really big sizes. Modern storage works differently, so if you really want to stream data from an SD card or something to the NES you need something more complex than a traditional mapper, capable of reading that data from the external storage and delivering it to the NES at the proper rate as the NES draws the picture.

Since VRAM on the NES is external (unlike most other consoles, which have VRAM built-in, the NES has VRAM/ROM in the cartridge), you can simply have your device/mapper monitor the requests the PPU makes while drawing the screen and deliver unique patterns and attributes for each 8x1 pixel area of the screen, without the NES side having to do any logic at all. This is the approach most people take when streaming video to the NES.
what if you made a cart that had its own processor on it, maybe something like a raspberry pi, and you actually tried to render out a fully 3D scene in real time, converted that to a video, and played it through the NES using the FMV system described above?
Obviously, there is no reason to ever actually do this.
It has been done. :mrgreen:
Sure, if you tried this same thing with the SNES you could have an image that was 256 colors in 15-bit color depth, and with that much detail you could render a modern PS4 title and still have it look pretty good.
The disadvantage with the SNES is that the cartridge doesn't have direct access to VRAM like on the NES, so you'd need an SNES program to actively transfer the video data provided by the external hardware to VRAM, and depending on how fast that can be done, the frame rate might take a bit. I don't code for the SNES, so I don't know how fast it can update a full color picture.

As for the color issues on the NES, one solution is to use RGB121, a simple RGB format that can display 16 colors using regular NES hardware. Here are a few examples of it:

viewtopic.php?p=94515#p94515
viewtopic.php?p=151067#p151067
viewtopic.php?p=151878#p151878
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: FMV on NES (dream I had)

Post by rainwarrior »

Well, for SNES there was the MSU1 homebrew expansion that has been put to use for FMV many times now. IIRC it started off as an emulator extension, but was later implemented on flash carts like SD2SNES.

Example: Super Road Blaster
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: FMV on NES (dream I had)

Post by lidnariq »

tokumaru wrote: Sat Feb 01, 2020 12:58 pm The disadvantage with the SNES is that the cartridge doesn't have direct access to VRAM like on the NES, so you'd need an SNES program to actively transfer the video data provided by the external hardware to VRAM, and depending on how fast that can be done, the frame rate might take a bit. I don't code for the SNES, so I don't know how fast it can update a full color picture.
The simplest way to send data to the SNES would be to upload a new palette and send as many 8bpp pixels as can fit in a vertical refresh, and extend blanking so that the amount of bandwidth consumed by the PPU balances the amount transferred in DMA.

The math works out to:
256x96 @60Hz / 256x120 @ 50Hz
256x144 @ 30Hz / 256x168 @ 25Hz
256x168 @ 20Hz / 256x200 @ 17Hz

Tearing will be visible at 20Hz - there just isn't enough memory for the S-PPU to hold one complete frame and the partial next frame.
User avatar
Kasumi
Posts: 1293
Joined: Wed Apr 02, 2008 2:09 pm

Re: FMV on NES (dream I had)

Post by Kasumi »

I am surprised no one has linked this: http://forums.nesdev.com/viewtopic.php? ... xt#p170357 (Cowboy Bebop NES)
Or this: viewtopic.php?f=2&t=11290 (Three short scenes)
So I will.
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Re: FMV on NES (dream I had)

Post by Bregalad »

Marscaleb wrote: Sat Feb 01, 2020 10:26 am Okay, so this idea literally came to me in a dream last night.
Amazing ! Welcome to the world of true NES-devers, having dreams (or nightmare) about NES development.
I imagine I'm not the first to think of it, but whatever.
Indeed. The earliest demo of the sort I can think of is the "motion" demo by Chris Cowell made in the late 1990s/early 200s.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: FMV on NES (dream I had)

Post by tokumaru »

There's also this idea I had a few years ago that went nowhere: http://forums.nesdev.com/viewtopic.php?f=21&t=13166
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Re: FMV on NES (dream I had)

Post by Bregalad »

tokumaru wrote: Sat Feb 01, 2020 5:09 pm There's also this idea I had a few years ago that went nowhere: http://forums.nesdev.com/viewtopic.php?f=21&t=13166
I think on the other side that your idea is the most realistic of the bunch. If we're going for a FMV on the NES it has to be realistic and take a reasonable amount of data for a reasonably long "video". If we just want to make as much as possible, then the rpi in a NES cart is the way to go. But if we want to use existing hardware such as MMC3/5 to do that, the approach of simply allow arbitrary tiles to be drawn on the screen is inssuficient, as this would fill the CHR-ROM very quickly as you pointed in your previous post.

Lowering the resolution by having 2x2, 4x2, 4x4 or 8x2 or whathever metapixels is easily feasible but will always look ugly, same with having larger borders.

So the only truly generic solution is to have some tileset with generic (but not blocky) geometric shapes and use it to draw all frames of the video, like if each frame was a normal NES "full-screen" picture, using only 256 tiles and a normal palette. This requires the total amount of colors per frame to be at bare minimum, so that's where anime cell-shading art style is coming to rescue. (this is largely used in anime even when there's no graphical limitations).
User avatar
Fisher
Posts: 1249
Joined: Sat Jul 04, 2015 9:58 am
Location: -29.794229 -55.795374

Re: FMV on NES (dream I had)

Post by Fisher »

Maybe if it was possible to swap some chunks of data to the CHR ROM/RAM from an SD card, some interesting animations could be created.
I'm probaly wrong, but I think the mapper would not be of much relevance in this case.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: FMV on NES (dream I had)

Post by tokumaru »

We know for a fact that modern technology can be put into cartridges to dynamically change the VRAM contents, directly feeding the PPU will all sorts of visuals, from pre-rendered video to hardware-accelerated 3D.

That's very interesting and all, but the NES is doing hardly anything at all. I don't know about you guys, but I personally am way more interested in what the NES itself can do. Instead of using an expensive cartridge to play an extremely blocky and dithered version of a full length movie that I could instead be streaming in 4K via Netflix using tech I already own, I'd very much like to see a normal (i.e. using simple, non-expensive tech) NES game adorned with cool little FMV cutscenes here and there. How far can we go in a normal NES setup? That's the real question for me.
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: FMV on NES (dream I had)

Post by Pokun »

I fully agree. Although I think it's neat to see it as a proof of concept, it's not much more than that unless it has some practical purpose. The fact that the PPU can be fed directly from the cartridge IS a feature of the NES, and I'm not against abusing it. Bank switching CHR-ROM for cheap animations is one way to use this feature.
Post Reply