Getting into 8-bit NESdev with only 64-bit free software

You can talk about almost anything that you want to on this board.

Moderator: Moderators

tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by tepples »

Sik wrote:You know, stupid question: what is Canonical's idea about what to do with 32-bit programs, wrap them in a VM automatically or expect users to run their own VM?
Going forward, free software should be recompiled for x86-64, and proprietary software should be packaged in a "snap" container.
Because I noticed sh happily passes the ball onto Wine whenever it's told to run a Windows executable, so it sounds like the wrapping could just be automatic.
In theory, it could be. If Ubuntu continues to offer Wine, there are two options:
  1. Package Wine as a "snap" with all its 32-bit dependencies
  2. Run only 64-bit Windows programs, which would require ensuring that FCEUX (Win32) and FamiTracker are 64-bit clean
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Sik »

Bregalad wrote:I never programmed any 8086 compatible machine in assembly, so I have absolutely no idea what you're talking about.
OK um:
  • 16-bit: real mode
  • 32-bit: protected mode
  • 64-bit: long mode
Real mode gave you 1MB of address space, split in 64KB segments (each segment being 16 bytes apart - yeah there was overlap, what it did was basically segment * 16 + offset). Opcodes were less orthogonal as well (e.g. you couldn't do LEA arbitrarily). There were four segment registers: CS (code), DS (data), ES (extra) and SS (stack). The ES segment was mostly to help with operations working across two segments :P (e.g. copying data)

Protected mode, besides making instructions more orthogonal and making the default register size 32-bit, gives you access to up to 32GB (though Windows likes to see only 4GB unless you mess with a registry entry) and provides a MMU with paging (which means a virtual address range can be mapped onto arbitrary non-consecutive physical ranges). A given segment may be up to 4GB long, and there are four rings (0, 1, 2, 3) each given different execution privileges. Segments could be code or data, and could be 16-bit or 32-bit (regarding executing code). Accessing an invalid address triggers General Page Fault aka what we call segfault. As you can imagine all this complexity means segments aren't addresses anymore, instead they're indices into a look-up table with all the segment properties, and it also meant that changing a segment register costs a very large amount of cycles (not to mention cache flushing). There were also two more segment registers (FS and GS).

...and most of that went ignored as usually only two rings would be used (for kernel and user space) and programs would normally just have three segments (code, data, stack) simulating a "flat" address space within each process and the segment registers only ever being touched by the kernel. Go figure.

Long mode gets rid of all that segment mess and just enforces a flat address space (aside from keeping FS and GS because Windows uses them to hold thread-specific data >_>). It also doubles the amount of registers (all, not just the integer ones) and extends the integer ones to 64-bit, and gets rid of some rarely used opcodes. Essentially it's a much lean version of protected mode that happens to also support 64-bit operations =P
tepples wrote:Going forward, free software should be recompiled for x86-64, and proprietary software should be packaged in a "snap" container.
Er, pretty sure you misunderstood both. Snaps are meant for all binaries (64-bit ones too), and they won't magically make 32-bit programs work out of nowhere if the kernel doesn't support it, which is the real problem from what I gather (hence the VM idea - it's literally about emulating those old programs). Only way snaps would work is if they actually included the full blown VM themselves and wrapped the program in it.

Also that won't help with stuff like DOSBox that still doesn't have a dynamic recompiler for 64-bit (so a 64-bit build is stuck with the slow interpreter), and I'm not sure how feasible is that. (on that note: somebody work on that please =P)
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by tepples »

Sik wrote:
Bregalad wrote:I never programmed any 8086 compatible machine in assembly, so I have absolutely no idea what you're talking about.
OK um:
  • 16-bit: real mode
  • 32-bit: protected mode
  • 64-bit: long mode
I thought at least somebody in this topic had referred to a 16-bit protected mode, which existed in the 286. Windows 3.x was designed for it.
Sik wrote:As you can imagine all this complexity means segments aren't addresses anymore, instead they're indices into a look-up table with all the segment properties, and it also meant that changing a segment register costs a very large amount of cycles (not to mention cache flushing). There were also two more segment registers (FS and GS).

...and most of that went ignored as usually only two rings would be used (for kernel and user space) and programs would normally just have three segments (code, data, stack) simulating a "flat" address space within each process and the segment registers only ever being touched by the kernel. Go figure.
Perhaps the cycle penalty is why the flat address space was chosen.
Sik wrote:
tepples wrote:Going forward, free software should be recompiled for x86-64, and proprietary software should be packaged in a "snap" container.
Er, pretty sure you misunderstood both. Snaps are meant for all binaries (64-bit ones too), and they won't magically make 32-bit programs work out of nowhere if the kernel doesn't support it, which is the real problem from what I gather
As I understood it, the kernel would still support both 32-bit and 64-bit programs, but the C library and other required userspace libraries wouldn't.
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Sik »

tepples wrote:I thought at least somebody in this topic had referred to a 16-bit protected mode, which existed in the 286. Windows 3.x was designed for it.
Which isn't present on the 386 or latter so it's not relevant to modern systems. I think it's safe to assume we talk about the 32-bit protected mode unless stated otherwise =P
tepples wrote:Perhaps the cycle penalty is why the flat address space was chosen.
It probably had more to do with the fact that all programmers wanted was just more memory. There were already real mode programs that behaved as if they were flat by doing arithmetic on segment:offset addresses (and why Intel couldn't go with their original plan of making segments not hardwired to 16 bytes granurality)

What it did influence though is discouraging microkernels. Worse, it's why Windows 9x is so unsafe: the penalty introduced by a syscall (which requires going into kernel space) would have been gigantic when you consider how many syscalls are called over time, so they just put everything in ring 0. Ouch. (NT didn't, but it was also slower)
tepples wrote:As I understood it, the kernel would still support both 32-bit and 64-bit programs, but the C library and other required userspace libraries wouldn't.
Then they wouldn't be talking about requiring running those programs from a VM (which you'd only do when the kernel is completely unable to run the programs on its own). I mean, Wine doesn't run 32-bit programs in a VM even though it provides all the libraries.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by tepples »

Sik wrote:
tepples wrote:I thought at least somebody in this topic had referred to a 16-bit protected mode, which existed in the 286. Windows 3.x was designed for it.
Which isn't present on the 386 or latter so it's not relevant to modern systems. I think it's safe to assume we talk about the 32-bit protected mode unless stated otherwise =P
I'm not sure what you mean. My 486SX ran Windows 3.1.
Sik wrote:
tepples wrote:As I understood it, the kernel would still support both 32-bit and 64-bit programs, but the C library and other required userspace libraries wouldn't.
Then they wouldn't be talking about requiring running those programs from a VM (which you'd only do when the kernel is completely unable to run the programs on its own). I mean, Wine doesn't run 32-bit programs in a VM even though it provides all the libraries.
The exact wording from the message was "snaps / containers / virtual machines". The first two are lighter weight than a full VM, more like a chroot or a Docker.
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Sik »

tepples wrote:I'm not sure what you mean. My 486SX ran Windows 3.1.
Which also ran in real mode by default, and supported 386's protected mode if you passed the /3 switch. (and Windows 3.11 always enabled it, effectively making 386 or later a requirement)

286 protected mode is a completely different beast from 386 protected mode and isn't supported on any CPU other than 286. Note that 386 protected mode can run 16-bit code (depends on what you put on the segment, and really all it did was just set the default size for 16/32-bit opcodes). It also had VM86 to run real mode code under a supervisor (as 32-bit Windows would do to run 16-bit programs). But it's not compatible with 286's protected mode.
tepples wrote:The exact wording from the message was "snaps / containers / virtual machines". The first two are lighter weight than a full VM, more like a chroot or a Docker.
OK reread it and you win =| Makes more sense.
User avatar
NovaSquirrel
Posts: 483
Joined: Fri Feb 27, 2009 2:35 pm
Location: Fort Wayne, Indiana
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by NovaSquirrel »

OSDev's page on 64-bit x86 CPUs says that in the 64-bit long mode, the compatibility mode for running 32-bit programs alongside 64-bit ones also supports 16-bit protected mode. This leads me to believe that support's still there, and has always been there 286-onward.
Joe
Posts: 649
Joined: Mon Apr 01, 2013 11:17 pm

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Joe »

All modern x86 CPUs are almost completely backwards-compatible with older x86 CPUs, including 286 protected mode. There are a few situations where things aren't compatible, and they fit easily into three categories:

1. Software that relies on undocumented behavior. Things like "pop cs" were never explicitly documented by Intel, so no one should have been using them in the first place.

2. Software that ignores the documentation. A lot of 286 protected mode operating systems put important data in areas that Intel documented as "reserved, must be zero". The 286 ignored that reserved data, but the 386 uses it to control 32-bit protected mode. This may be the reason why it seems like 286 protected mode doesn't work on newer CPUs - it's actually because the developers ignored the manual. (Software developers ignoring Intel's documentation is a common theme in PC history...)

3. Software that relies on behavior that should have been backwards-compatible, but isn't. I know of exactly one compatibility-breaking change to the x86 architecture. No one ever seems to bring up anything in this category when discussing x86 backwards compatibility issues.
lidnariq
Posts: 11429
Joined: Sun Apr 13, 2008 11:12 am

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by lidnariq »

Joe wrote:(Software developers ignoring Intel's documentation is a common theme in PC history...)
Not just software! 8259 IRQs 0 through 7 (interrupt 8-15) were configured to collide with bunch of CPU-internal faults, because interrupts 0 through 0x1F were supposed to be reserved for the CPU. (And the original 8088 didn't yet have anything above interrupt 4.)
User avatar
Myask
Posts: 965
Joined: Sat Jul 12, 2014 3:04 pm

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Myask »

Joe wrote:1. Software that relies on undocumented behavior. Things like "pop cs" were never explicitly documented by Intel, so no one should have been using them in the first place.
see also: LOADALL
Joe
Posts: 649
Joined: Mon Apr 01, 2013 11:17 pm

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Joe »

LOADALL is kind of a funny case. Intel documented it, but the documentation wasn't available to the public. Microsoft definitely had access to a copy of that document, which is why Microsoft's software only uses LOADALL when the CPU is a 286.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by tepples »

I have numbers. Numbers don't lie.
I love numbers almost as much as π.*

I just installed Xubuntu 16.04 (amd64) on a blank SSD. This puts me in an unusual position to tell exactly how much space support for 32-bit free software on a 64-bit system takes. First I installed some 64-bit free software useful for participating in NESdev:

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential gimp git python3-pil idle3 python3-numpy hexchat fceux retext
Then I ran this:

Code: Select all

sudo apt-get install wine
The result is three-fourths of a gigabyte.

Code: Select all

Need to get 192 MB/192 MB of archives.
After this operation, 735 MB of additional disk space will be used.
But at this point, much of this could be rendered moot by Mesen for GNU/Linux.


* Yes, π is a number too.
User avatar
Sverker
Posts: 46
Joined: Wed May 23, 2012 11:30 pm

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by Sverker »

I don't see the point of principles like "free/open source software only" except for the sake of posturing, and at that point you've already lost the patient. It just seems so needlessly self limiting, but maybe I'm just utilitarian like that.
calima
Posts: 1745
Joined: Tue Oct 06, 2015 10:16 am

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by calima »

You cannot tell what a binary blob does, it is a security nightmare that could do anything from cryptolocking your files to installing Windows 10.
tepples
Posts: 22705
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Getting into 8-bit NESdev with only 64-bit free software

Post by tepples »

You could do the same thing you do with proprietary NES games: contain them. Once contained, proprietary software isn't substantially more dangerous than underhanded free software. If proprietary software can't write outside its chroot, for example, it can encrypt only those files that get served writably into its container. And the container's administrator can set a quota for /home so small that the only way Windows 10 gets installed is by adding a decimal point.

I still don't see how 32-bit free software, such as FCEUX (debugging version) in Wine in multiarch Linux, is likewise "a security nightmare".

I run mostly free software on my laptop, both 32-bit and 64-bit, with a small set of carefully chosen exceptions. A few weeks ago, I replaced my laptop's HDD with an SSD and installed a fresh copy of Xubuntu. So I decided to keep a list of all programs that I have installed. As far as I can tell, only these proprietary programs are installed on my laptop's SSD:
  • The firmware that bcmwl-kernel-source installs
  • Dropbox and Skype, needed to communicate with clients
  • A set of freeware fonts, including Microsoft core fonts (which run either in a bytecode emulator or with autohinting)
  • Scripts in HTML documents, which are restricted by the browser's sandbox, the same-origin policy, and Disconnect's blacklist
  • Programs that I make that haven't been distributed to the public yet
  • Programs that I make for clients, which must be proprietary because of industry constraints
  • Programs made by other forum members, which I am evaluating
Post Reply