Page 2 of 2

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 9:38 am
by gauauu
tepples wrote: NES homebrew community in general
While I have sympathy for people with poor internet connections, that isn't a problem that you are REQUIRED to solve for them. But if you WANT to consider them, then providing a compiled exe is usually the smallest thing. If it's worth it to you to invest your time converting from python to C in order to provide those people a tiny binary to download, then you've answered your question: yes it's worth it. Release a compiled binary version of your tools.
Retrotainment Games
I'd ask them what they personally would want. Explain the trade-offs, and how much development time it would take you, and just ask them what they prefer. The 20 seconds of rebuilding might be annoying to them, but maybe it's worth it to let you focus your development time on the game itself. Now that they already have python set up, do they hate dealing with it and want you to change it? Or do they just want you to buckle down and get working on Full Quiet? The only way to know is to ask them.
Do binary release platforms generally support this? Or do they require all artifacts to be built from the same tag?
There are quite a few different binary release platforms, but in general, they're pretty flexible.
Eventually, the public
"The Public" as a general category is really hard to plan for. Because that can be anyone from my mother-in-law on a chromebook, to Tokumaru. Supporting multiple platforms well is a LOT OF WORK. You have to decide what subset of the public it's worth supporting. There's no silver bullet.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 9:43 am
by gauauu
Would it be better to make two separate repositories for each game: one for the game's source code and assets and one for its asset conversion tools?
If you're allowed to make the tools open source, then there's a lot more flexibility with pricing on build servers. Appveyor, for example, is free for open source projects.

So if that's the case, having an open-source tools repo and a closed source game repo might make sense.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 12:46 pm
by slembcke
Eh. I get pretty annoyed when I need to install yet another copy of visual studio on my Windows machine for some project. That crap is huge. The last contract I worked on targeted a couple of consoles and I cleaned up almost a hundred GB of SSD space deleting those tools. Ridiculous. I get why people don't like installing "yet another tool".

I tend to stick to Python (2) and plain C/Make depending on what I'm doing. Installing both is dozens of megabytes of disk space. A fraction of one of those ridiculous Electron apps. I don't feel too bad about using smaller tools like that as a requirement.

Why Python 2? It comes installed on Macs and Linuxes. Does anything ship with Python 3? I don't even know (or, quite honestly, care) what the difference with 3 is. I get the impression it's a lot of language pontification to me about a language that's already a little haphazard.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 1:33 pm
by rainwarrior
Okay, so this isn't just about Python, then.

It seems like you have your standard setup that requires:
  • GNU Coreutils
  • GNU Make
  • Python
  • Pillow
  • ca65
And you've gone far and above most open source projects with a cross-platform environment setup guide:
https://github.com/pinobatch/nrom-templ ... /README.md

So, if you want commentary on this... I guess personally I would have little problem setting this up, especially with these directions, but honestly it's a few steps beyond what I'd want to undertake casually. If I had a need to build one of your things, I'd definitely be willing, but admittedly you've put up an unpleasant barrier here. As it stands, I've often looked at the source code for things you've published, but the only time I've actually set up to build something of yours was when I was getting paid to do it for Haunted Halloween 85.

First and foremost this is obviously Linux-centric. Except for CC65, everything here is easy to get or very standard tools on Linux. Not so on Windows.

One thing I would immediately suggest is not to ask any users to manually modify their PATH, ever. This should simply not be done, except by an installer for a big well maintained stable program. Having to add CC65 to the PATH is a non-starter for me, I refuse to make it a global part of the system. I personally know how to create a temporary PATH in a command environment, but it'd be an annoying work-around. Python on the path is fine. MinGW / MSYS is maybe OK (though they recommend against it, see below). CC65 no.

GNU Coreutils / GNU Make are not very standard on Windows. You probably have them if you're a Windows programmer who needs to build a lot of Linux-centric open source software that doesn't try to make concessions for Windows, but otherwise these just aren't standard toolkit for Windows programming. I don't know what you require from Coreutils but it might be worth trying to drop it as a dependency. Make is less obscure for Windows than Coreutils, but most Windows-centric development environments will do without it.

Your install process directions for these two breaks down entirely. You give a few options:
  • You mention MSYS but don't give a link to download. To be honest, I'm not entirely sure what MSYS is, I just think of it as a component of MinGW. The primary website for it seems to be this one which is an absolute nightmare to read.
  • You mention Git for Windows, and then some hideous process to add Make. Also not a good solution for this problem. (Maybe you'd be surprised but I think most Windows users of Git are doing it through a GUI like Sourcetree that keeps a local copy of git rather than a globally installed command line tool.)
  • You mention devkitPro, which I think is an awful place to get these tools. devkitPro's wiki doesn't even have a clear description of what it is or what it's for. It seems to be about homebrew development for Nintendo consoles GBA and later? This is not even remotely something I'd want to trust to do global installation stuff on my system. Maybe if I was already making homebrew for these other platforms and was familiar with it, but this is too obscure looking for me to be confident its installer won't mess up my system.
I think what you should really recommend to install for this is MinGW. It's well known, well maintained, and has a relatively nice installer. Only small problem is it doesn't install to your PATH (oops). That's actually good though, because I wouldn't want it to install to my path. I do use MinGW from time to time, but I much prefer keeping it in its own isolated environment. The MinGW Getting Started guide also recommends this:
The MinGW team do not recommend modifying the system wide Windows PATH variable. We prefer that you use a script to set PATH for the processes on a per session basis.
Alternatively, if you can ditch coreutils the problem becomes much simpler. Just get Make for Windows. All you need from this is a single EXE. You can drop it locally in the folder. Done. (I think for HH85 I used this + some small modifications to the makefile to get around whatever is in Coreutils.)

Okay, so that was the worst of it.

Python, I think has a mostly easy install process on Windows these days, not going to further comment on that.

Installing Pillow is OK. You need to open a command line, but at least its a command that should reliably work for most people, provided they know what a command line is. Unlike a lot of other Python packages, Pillow has always installed correctly via PIP for me, and I think it's well maintained for that.

Finally CC65. I've already said this above, but don't require this to be a global install, i.e. do not ask users to set their PATH. Use a local version of it.

Personally I have a lot of CC65 projects, and I use a local copy of CC65 in every one of them. CC65 is not rigorously maintained to be continously backward compatible, and there have been several incidents of this. If I'm working on something I won't allow it to be disrupted by an unnecessary update to the tool.

So...

All of that said, if you have a user install MinGW, there's two approaches that I think might be easy to set up that missing part of the environment:

1. Run C:\MingGW\msys\1.0\msys.bat to open the MSYS shell, which will have access to Coreutils and Make, and they can run this pseudo-Linux environment to build your thing. (I guess put it in your msys home folder.)

2. Create a batch file that sets its own temporary PATH before running Make locally.



So... this turned out pretty long, eh... sorry there was so much to say, but maybe it will help understand what it looks like for at least this particular Windows user to consider building one of your projects.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 1:37 pm
by rainwarrior
slembcke wrote:Does anything ship with Python 3? I don't even know (or, quite honestly, care) what the difference with 3 is. I get the impression it's a lot of language pontification to me about a language that's already a little haphazard.
Many Linux distros ship with both 2 and 3. What has been the most important improved feature for me in 3 is much better support for Unicode across the whole language and toolset, but beyond that there have been a ton of small and worthwhile additions to the Python standard library. Python 2.7 is just 2.7 on life support since 2008, which has its own nice appealing stability I suppose.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 2:10 pm
by Sour
rainwarrior wrote:Maybe you'd be surprised but I think most Windows users of Git are doing it through a GUI like Sourcetree that keeps a local copy of git rather than a globally installed command line tool.
I used to use SourceTree when I started off using Git, but nowadays the little that Visual Studio doesn't support on its own, I just do in a command prompt (I could very well be the minority here, though.)

Generally speaking, though, I agree. I would usually never bother installing that much software just to compile something, unless I desperately needed it. As someone who develops using VS, C# is my go-to language for any small utility I want to make, so I don't even have Python installed.

Also, regarding adding programs to the PATH. Having a PATH variable that is too long (in terms of the number of characters in the string) can break a lot of stuff - though I don't quite remember the specifics (it may have been fixed in Win8 or 10). In Win7, it's definitely been a problem for me in the past at work (due to having multiple versions of SQL Server installed, each of which adds quite a lot of stuff in the PATH var). So yea, it's usually best to avoid adding stuff to the PATH variable unless it's really necessary.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 2:15 pm
by slembcke
Is there a particular reason to avoid Cygwin? I've had a few people respond with utter revulsion to the idea of it, but it makes it pretty trivial to install most unix tools painlessly in a couple minutes including GCC, Make, Python, etc. It even puts it everything in a single directory so you can clean it up easily.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 2:32 pm
by gauauu
All those things considered, if I was part of a team (ie retrotainment) that already had a set of tools installed and working, I'd want to discuss it as a team before making changes to that workflow. Maybe it's worth changing, but maybe they'd just want to leave their build flow alone, not touch it, and keep working.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 2:47 pm
by rainwarrior
slembcke wrote:Is there a particular reason to avoid Cygwin? I've had a few people respond with utter revulsion to the idea of it, but it makes it pretty trivial to install most unix tools painlessly in a couple minutes including GCC, Make, Python, etc. It even puts it everything in a single directory so you can clean it up easily.
Cywgin slipped my mind as another alternative. To be honest I haven't used it in over a decade, so I don't have any current experiences to offer as comparison.

Is there anything you can think of about it that's better or worse than using MinGW/MSYS for this purpose? MinGW is something I do use, and think it's usually pretty good, has a relatively nice installer, and keeps itself nicely contained. (Related: does anyone know what MSYS2 is for?)

Though as I suggested above, probably a more ideal solution would be to ditch this whole Linux toolchain dependency, and just provide make.exe and a batch file to set a temporary PATH and run Make.


...and just a reminder, all of this is just suggestions on what to do if you want to make your Windows user experience better. You don't owe free maintenance to anybody, and while the current setup is unpleasant it's not terribly unreasonable. I've had much worse times building much higher profile open source projects than these. :P The only red flag here for me is that you're telling users to set their PATH, which I think is bad.

Re: Porting my tools to C or C++

Posted: Tue Feb 19, 2019 10:40 pm
by Oziphantom
I have rarely needed to install 2 VS at once ( the 2008 version was a bit special though ). On my current 2017 install I've built VS6 projects for some old tools. I have to fix the modern security warnings but, not a problem.(MS actually have a help page dedicated to the "so you want to build something from Win95, well you're going to get this bug, fix it by changing this, toggle this option and then add this line" handy dandy guide) .This is why VS is this massive gigantic blob, it does support and kept support for these things.

Also I would point out that I'm talking about small 1, 2 file command lines tools, and tepples making them Windows safe, not actually giving a MSVC sln file. As a practical example of what I'm talking about, here are some real world examples, https://sourceforge.net/p/tass64/code/HEAD/tree/trunk/ this is probably a limit, however Soci is a pure no windows guy, when he tests windows he tests in reactOS, however this code is very pure and with a couple of minor fixes i was able to get it compiling on MSVC. Soci was then willing to make said changes to his code base and now I can just grab his code, make a "solution with existing files" remove the DOS and AMIGA custom stuff and done.

Another smaller example is https://github.com/bitshifters/exomizer ... 2/rawdecrs (not original author as it is code is only in a zip ) simple small C files to do a thing, again throw it into a sln with source and done.

Cygwin is a name that gives me nightmares, it did some horrible stuff to my machines back in the day, might be fine now.

I think another path to this would be
install virtual box
download lubuntu.iso
make lubuntu vm
follow instructions as per normal

(although if anybody has a good guide on how to set up cross building on linux, that would be great thanks. Everything I've read has been here is one tiny step and we assume you understand the other 50 archaic linux commands to patch it all up as you need )

I would also vote that CC65/CA65 is somewhat Linux centric and seems to be a bit of a pain to use on windows. Possible, but I get the impression it really wants a linux host.

I use the github desktop for my git, which has its own isolated git, and when you go to its menu and say "open git command line prompt" it will bitch that it can't find git :D

Re: Porting my tools to C or C++

Posted: Wed Feb 20, 2019 8:24 am
by gauauu
We're drifting a little off-topic from the original post, but here's one thing I've done to address the windows/linux build stuff:

My development machine, and my entire build ecosystem and tools are linux-based. When I want to give a windows collaborator the ability to compile things, instead of making a windows version of the build, I use Vagrant. If you aren't familiar with Vagrant, it's a tool to manage a local virtual machine specifically for development. I have a Vagrant script that knows how to setup and create a linux vm that can do all the builds. The windows user needs to install Vagrant (and a vm program like virtualbox). Then they just run a "build.bat" script that I gave them, which ensures that the vm is running, and asks the vm to run the build process.

It's incredibly slow the very first time (it has to download the vm image and set up the vm, which can take 5 minutes), but reasonably fast after that, and requires very little fiddling about on the windows side.

There are obvious disadvantages (a vm is slower and heavier than running tools in the host OS, and requires a huge image download), but it's nice being able to provide a windows user with the ability to build without having to set up a fiddly development environment. Installing Vagrant is generally easier than fiddling with multiple languages and tools and paths.

Re: Porting my tools to C or C++

Posted: Thu Mar 14, 2019 1:00 pm
by tepples
As I mentioned in an aside to this post, I'm ready to discuss deprecating recommending that users modify the system-wide PATH.

As I understand it, "ditch Coreutils" means that if the script detects Windows (through the presence of the COMSPEC environment variable), it should not try calling cat or rm. Instead, it should call cmd /c copy /b (with + signs between each source file path argument and the next) or cmd /c del instead. Were the "small modifications to the makefile" of this nature?

Would it be a good idea to include such a "batch file that sets its own temporary PATH before running Make locally" with each of my projects? The user would still have to edit each file depending on where in the file system he or she unzipped cc65 and Make.

What's the best way to work around the likelihood that the user will have spaces in pathnames? Examples include C:\Program Files (x86) or C:\Users\John Doe. Is there a prerequisite-driven build system that's preferable to GNU Make on the Windows platform, such as with better support for spaces in pathnames and the requirement of a plus sign between source arguments to copy /b?

I'm also not certain of the most appropriate way to implement "use a local copy of CC65". I don't expect that you meant check a vendored copy of the cc65 source code into the repository. Or some procedure to copy ca65 and ld65, and if necessary cc65 and the C runtime library, into a subdirectory that's specified in .gitignore?

Re: Porting my tools to C or C++

Posted: Thu Mar 14, 2019 1:52 pm
by rainwarrior
tepples wrote:As I understand it, "ditch Coreutils" means that if the script detects Windows (through the presence of the COMSPEC environment variable), it should not try calling cat or rm. Instead, it should call cmd /c copy /b (with + signs between each source file path argument and the next) or cmd /c del instead. Were the "small modifications to the makefile" of this nature?
I think the implied question in "ditch Coreutils" is why does your makefile need to use cat or rm at all? I don't know what you expect a makefile to do, but probably it should be doing as little as possible. Can't you just build binaries to the location they're needed? Why is there some extra layer of OS level file manipulation on top of it? Maybe try not to do that? (Or if you really must... maybe one of your other, harder dependencies can handle it. Python maybe?)
tepples wrote:Would it be a good idea to include such a "batch file that sets its own temporary PATH before running Make locally" with each of my projects? The user would still have to edit each file depending on where in the file system he or she unzipped cc65 and Make.
That would be an improvement I guess. If CC65 was local to the project they shouldn't have to edit any paths?
tepples wrote:What's the best way to work around the likelihood that the user will have spaces in pathnames? Examples include C:\Program Files (x86) or C:\Users\John Doe. Is there a prerequisite-driven build system that's preferable to GNU Make on the Windows platform, such as with better support for spaces in pathnames and the requirement of a plus sign between source arguments to copy /b?
That's not really a problem unless you're using absolute paths for things? If your project has no spaces in its directories, it won't come up in a relative path. Where is this actually causing a problem in whatever you're thinking about doing?
tepples wrote:I'm also not certain of the most appropriate way to implement "use a local copy of CC65". I don't expect that you meant check a vendored copy of the cc65 source code into the repository. Or some procedure to copy ca65 and ld65, and if necessary cc65 and the C runtime library, into a subdirectory that's specified in .gitignore?
In all of my projects I just tell people to unzip CC65 into a specific folder. I don't know what could possibly complicate that, and I'm not willing to go down whatever rabbit hole you're drawing in with the word "vendoring".

Re: Porting my tools to C or C++

Posted: Thu Mar 14, 2019 2:13 pm
by tepples
rainwarrior wrote:why does your makefile need to use cat or rm at all?
cat or copy /b: At one point, I used to build iNES header, PRG ROM, and CHR ROM to separate files rather than adding CHR ROM as a MEMORY area and SEGMENT in the linker configuration. I'm definitely not using copy /b to combine the header and PRG ROM anymore, and I don't think I'm still doing that for INES+PRG ROM and CHR ROM anymore, but I'd have to check all my repos to make sure of that.

rm or del: I currently use rm for make clean.
rainwarrior wrote:[A batch file that sets PATH] would be an improvement I guess. If CC65 was local to the project they shouldn't have to edit any paths?
Someone building one of my projects would have to edit the path to GCC in order to build cc65 from source code, making MinGW a dependency on Windows. Or he or she have to download and unzip Make and cc65 built for Windows x86 or x64 into a particular subdirectory (listed in .gitignore) and then repeat the process for each other project of mine that he or she wants to build.
rainwarrior wrote:That's not really a problem unless you're using absolute paths for things?
The path to Make and cc65 on the user's system would have absolute paths unless I instruct the user to copy Make and cc65 into a subdirectory of each project.

To others reading this: What are the drawbacks of instructing the user to copy Make and cc65 into a subdirectory of each project?

Re: Porting my tools to C or C++

Posted: Thu Mar 14, 2019 3:08 pm
by koitsu
tepples wrote:To others reading this: What are the drawbacks of instructing the user to copy Make and cc65 into a subdirectory of each project?
That I can think of, very few. The main two I can think of are disk space and "technical debt". Disk space isn't a big concern here because the programs are pretty small, while the latter usually encompasses many caveats: but in this specific case (of Make + cc65) the only debt I can think of would be the administrative maintenance of updating the binaries if/when new releases of software comes out.

I think those "drawbacks" are pretty simple/easy (thus acceptable and reasonable) compared to, say, requiring a person to install Visual Studio.

The "keep everything in a local directory" approach is honestly better than screwing with %PATH% for several reasons, the biggest being that introducing a new "commonplace name" binary/program into %PATH% trickles down into literally everything and you can end up with painful binary conflicts. make.exe is a great example: say you have D:\bin\make.exe (the gnuwin32 version), but you also run Cygwin or MinGW which has their own make. In Cygwin or a MSYS2 shell you type make -- and you end up getting the wrong one. This isn't just limited to make either. Yes, this has to do with %PATH% and $PATH argument order and the like, but it's a serious problem if/when you encounter it (I have many times) with all sorts of utilities. It's why I don't use Cygwin or MSYS2 unless I have to (and if I do, I always pick MSYS2). Keeping everything in the project's directory "mostly" relieves this.

P.S. -- All of those utilities on the gnuwin32 Sourceforge page are horribly old and are often fragile compared to MSYS2 or Cygwin alternatives (both of which are actively maintained), all the way down to not having good UTF-8 support (including in filenames; though Windows 7 native CLI utilities often botch these too). I do have several gnuwin32 binaries that I keep laying around in my D:\bin directory, and I fight with/scream at them every time I have to deal with them. wget is the most notorious. The way they break is often weird, too. I am not saying don't use them, I'm just saying if/when you encounter oddities with them, a lot of the time (in my own anecdotal experience) they tend to be due the age/how they were built/whatever with no real way to solve the problem other than ditching them. I tend to just fall back on my FreeBSD or Linux boxes to do things if they don't work for me. It's a bummer, too, because I really love how "standalone" the gnuwin32 programs tend to be. :/

I really do urge folks to give MSYS2 a try if they've found Cygwin to be bloated, slow, or just generally don't like it -- it even gives you an actual terminal (mintty) with good terminal support alongside bash. Their wiki page goes over a lot of stuff. MingW fits into it quite well.