Syntax highlighting added to Wiki

Discussion about the site's wikis, including bugs/issues encountered.

Moderator: Moderators

User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

I will try to make the time to undo this come the weekend. Remember: I have many, many wiki documents to manually edit by hand any time this thing is enabled or disabled.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

koitsu wrote:I will try to make the time to undo this come the weekend. Remember: I have many, many wiki documents to manually edit by hand any time this thing is enabled or disabled.
Well, I might as well try to fix the problem before that. It needs a modification in the SyntaxHighlighter scripts.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

thefox wrote:
koitsu wrote:I will try to make the time to undo this come the weekend. Remember: I have many, many wiki documents to manually edit by hand any time this thing is enabled or disabled.
Well, I might as well try to fix the problem before that. It needs a modification in the SyntaxHighlighter scripts.
The attached script should fix the issue. Replace the <script> include of shCore.js with an include of this shCore-nesdev.js. I also updated the brush slightly.

Test case: http://wiki.nesdev.com/w/index.php/Scan ... rge_Tables
Attachments
shCore-nesdev.zip
(21.3 KiB) Downloaded 644 times
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

Thank you! I'll take care of this later tonight after work.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

Okay, this is all extremely complex, with many things that could go wrong any moments, so I have opted not to do this yet. Let me discuss the situation.

1. ALL CHANGES THAT DIFFERENTIATE FROM STOCK SOFTWARE MUST BE DOCUMENTED/PATCHES/DIFFS MADE. I do this any time I make a change; this responsibility falls on me, but minimising the number of diffs from stock software is important. Failure to do this results in basically 100% inability for ANY kind of software upgrades to reliably be done. Rephrased: the more one-offs we have to deal with, the more painful upgrades (of MediaWiki, MW extensions, forum software, whatever) become. I think Tepples knows bits/pieces of what the situation is, and I know other bits/pieces.

At this time we already have one diff for SyntaxHighlighter, and I will explain what it contains (two things):

i) Modifies SyntaxHighlighter.php to include "6502" and "asm" SyntaxHighlighter engine references, which refer directly to the engine called "Assembly6502".

ii) Installs a new file, scripts/shBrushAssembly6502.js, which is quite small (1.9KBytes). Your .zip file contains a replacement that appears to be almost identical to this (but not entirely), but the filename is called shBrush6502.js so I would need to rename it (not a problem).

2. The default shCore.js file is 16175 bytes. The shCore-nesdev.js file in the .zip you uploaded is 72517 bytes. This is enormous for Javascript, by the way. After a review of it, it looks to me like it needs to be packed or what's otherwise known as "minified". I have no idea how to pack anything related to Javascript (edit: maybe this is helpful), but there's no way we're putting a 72KByte JS file into place. Minified should be significantly smaller.

3. The author of SyntaxHighlighter, on his site last month, stated that he's working on maintaining the thing once more. However no where in github does he provide the built versions (and I don't know if they're packed/minified).

I won't be changing anything to "include shCore-nesdev.js rather than shCore.js" because that just creates more one-offs. Instead I'll just have to name the thing shCore.js, once a packed version can be provided.

Don't worry about the comments at that Javascript packing site about gzip or webserver compression; that's stuff that has to happen within Apache/on the server side and would require WhoaMan to make changes (it helps substantially). What I used on Parodius was below (and greatly relied upon mod_mime_magic to help properly detect things too):

Code: Select all

LoadModule deflate_module libexec/apache22/mod_deflate.so
LoadModule mime_magic_module libexec/apache22/mod_mime_magic.so

<IfModule deflate_module>
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE text/csv
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE application/javascript
  BrowserMatch "^Mozilla/4" gzip-only-text/html
  BrowserMatch "^Mozilla/4\.0[678]" no-gzip
  BrowserMatch "\bMSIE" !no-gzip !gzip-only-text/html
</IfModule>
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

2. The default shCore.js file is 16175 bytes. The shCore-nesdev.js file in the .zip you uploaded is 72517 bytes. This is enormous for Javascript, by the way. After a review of it, it looks to me like it needs to be packed or what's otherwise known as "minified". I have no idea how to pack anything related to Javascript (edit: maybe this is helpful), but there's no way we're putting a 72KByte JS file into place. Minified should be significantly smaller.
Meh, I agree that 70 KB is on the large side, but the JavaScript files typically are static, so the browser will download them once and cache them for a good time. More often than not, the browser will also accept them gzipped.

The reason I didn't compress the JavaScript was that the first packer that popped up on Google (http://dean.edwards.name/packer/) broke the script.
I won't be changing anything to "include shCore-nesdev.js rather than shCore.js" because that just creates more one-offs. Instead I'll just have to name the thing shCore.js, once a packed version can be provided.
Feel free to do whatever you want with the files.

Anyway, I guess I should tell you how shCore-nesdev.js was (manually) built:
1) Started with src/shCore.js from the distribution ZIP.
2) Took the file scripts/XRegExp.js from the repo (https://github.com/alexgorbatchev/synta ... 83/scripts) and pasted it on top of the shCore.js file. This dependency was missing from the distribution ZIP, but was probably added to the packed shCore.js file in the build process.
3) Applied my modifications to the shCore source. You can find them from shCore-nesdev.js by searching for "thefox". It's only 3 lines.

There are some build scripts in the SyntaxHighlighter repo to automate the packing/etc, but I couldn't bother to install PHP to run them.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Syntax highlighting added to Wiki

Post by tepples »

koitsu wrote:the more one-offs we have to deal with, the more painful upgrades (of MediaWiki, MW extensions, forum software, whatever) become. I think Tepples knows bits/pieces of what the situation is, and I know other bits/pieces.
I've tried to help with this by dividing LocalSettings.php on the wiki into two parts: the standard stuff that comes with vanilla MediaWiki above the line, and site customizations (extensions, etc.) below. And I've tried to put a date on each substantial change to LocalSettings.php. Is there something else that I should have been doing?
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

So here's the status:

1. I've put things into place verbatim, and they do seem to work. That said:

2. I had to Shift-Reload to pick up the .js changes. I was a little surprised by this, given that the file sizes differ substantially (old vs. new) as well as file timestamps, so I'm not sure what's going on there. I think MediaWiki does offer some way to force a cache update for clients, but I don't think that'd affect this because its not the MediaWiki page content itself that changed, only the stuff referenced by <script> (the filenames haven't changed, etc.). That should all be server-side/Apache-side as far as HTTP server vs. client headers go.

3. All the packers I tried did in fact break the Javascript as you said. Some got them down to reasonable sizes (17KBytes), but they'd break with syntax errors / missing semicolon and the like. As such, I'm going to put in a request to WhoaMan to enable mod_deflate (either for all of Apache or just within the nesdev VirtualHost contexts) to rectify the situation. It's the best I can do.

The reason the file size matters so much is both an issue of wasted network bandwidth and overall speed (it now takes 6x longer to download that single .js file, plus the time it takes to parse it/run it on the client). 76KBytes of JavaScript probably isn't going to make things like tablets (ex. Kindle devices) and other embedded devices very happy. I'm crossing my fingers though. If we get complaints, I'm pulling the entire syntax highlighting crap out entirely and will manually go through all the pages and remove the tags (this takes me an hour plus to do).

I think it's utterly ridiculous how crappy all these syntax highlighters are. I keep seeing new ones pop up on the web all over the place on blogs and so on, but god only knows what's going on under the hood there.
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

tepples wrote:
koitsu wrote:the more one-offs we have to deal with, the more painful upgrades (of MediaWiki, MW extensions, forum software, whatever) become. I think Tepples knows bits/pieces of what the situation is, and I know other bits/pieces.
I've tried to help with this by dividing LocalSettings.php on the wiki into two parts: the standard stuff that comes with vanilla MediaWiki above the line, and site customizations (extensions, etc.) below. And I've tried to put a date on each substantial change to LocalSettings.php. Is there something else that I should have been doing?
No I think that's correct (I follow your same methodology when it comes to LocalSettings.php), but when it comes to extensions I just wanna make sure any hand-modifications to things get backed up as diffs and so on.

I put everything in /www/vhosts/nesdev.com/wiki/mediawiki_extensions/ as you've probably noticed, and fix perms (664 not 644, since group-write (apache) is the only way we can modify each other's stuff. For example your Cite.tar is 644 right now).

For the file permissions stuff, just stick umask 0002 (default is 0022) in your .bashrc on the server and that should take care of new file creation permissions being correct going forward.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

koitsu wrote:3. All the packers I tried did in fact break the Javascript as you said. Some got them down to reasonable sizes (17KBytes), but they'd break with syntax errors / missing semicolon and the like. As such, I'm going to put in a request to WhoaMan to enable mod_deflate (either for all of Apache or just within the nesdev VirtualHost contexts) to rectify the situation. It's the best I can do.
If you want to try, there's a packer written in PHP in the build directory of Syntax Highlighter repo at https://github.com/alexgorbatchev/synta ... ee/3.0.83/. This was probably used to pack it up for the official release, so it may not break it.
76KBytes of JavaScript probably isn't going to make things like tablets (ex. Kindle devices) and other embedded devices very happy.
The parsing/running could very well be even slower if the file is packed, depending on what kind of packing is done. Minifying (white space stripping) is probably the best option.

Now I got a little bit curious why the minifying/packing was breaking the script, so I may check out what's wrong at some point.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

OK, I think I found the problem. A semicolon was missing on one of the lines (gotta love JavaScript's automatic semicolon insertion). The attached file should fix the problem. It's not compressed or minified, feel free to do that with whatever you want.
Attachments
shCore-nesdev-2.zip
(20.41 KiB) Downloaded 639 times
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Syntax highlighting added to Wiki

Post by tepples »

Bump again: it just came up in #nesdev.
In #nesdev, thefox wrote:The frame and NMIs
the javascript syntax highlighting is messed up on this page, you should be able to hack in a fix among the previous hacks by editing shCore.js
"messed up" meaning :+ gets converted to ;:+
I'm not familiar with the syntax highlighting codebase; koitsu, are you still around?
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

You can search for "thefox" in shCore.js, there are a bunch of replace() lines. If you add "&#160;" -> " " in there, that should take care of the issue. " " should preferably be a non-breaking space, as that's what the HTML entity 160 stands for.

Unfortunately this would be, like the other existing replace() lines, just another hack, because all HTML entities are not covered.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Syntax highlighting added to Wiki

Post by koitsu »

I'm still around. I said I'd get to this when I can. And I will. I just have to go through all the pages I added to make use of the syntax highlighter and remove the syntax highlighting bits by hand, followed by removal of the syntax highlighter altogether.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Syntax highlighting added to Wiki

Post by thefox »

Eh, that was not the point. The bug can be fixed by applying the small modification I mentioned in the previous post.

If you still want to remove the syntax highlighting, though, that's fine by me.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
Locked