Page 1 of 1

Assessment for Judging

Posted: Mon Feb 19, 2018 9:42 pm
by M_Tee
Being a teacher, assessment methods (rubrics, grading scales, and the like) are all of particular interest to me.

How are you each approaching judging?

What factors are you including in the overall categories, etc.?

Re: Assessment for Judging

Posted: Tue Feb 20, 2018 8:17 am
by na_th_an
It's hard to tell. I usually begin from top to bottom. I usually pick up muy most favourite in each category and give it full marks if I think it deserves it, or almost full marks. Then I pick up my next favourite, and measure how much "worse" it is so I give it the same, 1 less or 2 less or ...

When I'm done I reconsider my decisions. Maybe I was too harsh with my least favourite. Maybe the entries awarded the less points should have higher marks.

So it is quite relative. I don't know if this is a good or bad method, but it's the method I use most of the time in every task where I need to qualify or simply pick an arbitrary order to stuff.

On top of that, I think each one has a different approach to what topic or category is most important. For example, I don't care if a game is original as long as it is fun. Well, I do care that it is original, but that's less important. I don't care if it's technically top-notch as long as it looks good and plays great, and feels polished and complete.

Re: Assessment for Judging

Posted: Tue Feb 20, 2018 8:47 am
by FrankenGraphics
One way, though it requires a bit of prequisite legwork, is to set up atomized and universally functioning merits, and assign them a value (nominally 1). Each merit which you find true rewards a point (or a half, or whatever value you've assigned to that merit). Since some categories can reward more points than others, you'd either have to come up with more merits or attribute them with more points each.

Ie.
the apple was:
-rinsed (1pt)
-good for baking (1pt)
it had:
-an appetizing colour (1pt)
-a nice scent (1pt)
-a distinct flavour (1pt)
-a nice texture (1pt)

Maybe if the apple had a really nice scent, it might be worth 2pts.

Or the merits could be accumulative, perhaps like this:
-the student has grasped the elementary concepts needed to pass the course and continue with the next (threshold 1pt)
-the student has been able to reflect on the concepts and relate them to other related fields of study (threshold 3pt)
-the student has achieved a confident and profound understanding of the subject. (threshold 5pts)

Re: Assessment for Judging

Posted: Tue Feb 20, 2018 8:59 am
by dougeff
I ordered them by how much I like them, then adjusted their scores until the numbers came out the way I wanted. Unfair? maybe.

I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.

Re: Assessment for Judging

Posted: Tue Feb 20, 2018 10:06 am
by gauauu
dougeff wrote: I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.
I think it was mentioned somewhere that we should be rating them based on the version available at the deadline/submission, and not based on later improvements, so I agree with this.

Re: Assessment for Judging

Posted: Tue Feb 20, 2018 2:22 pm
by na_th_an
That's why I haven't still commited to introduce any of the enhancements / corrections suggested for my entry. I will do when the results are posted, for the cart.

Re: Assessment for Judging

Posted: Wed Feb 21, 2018 8:55 am
by toggle switch
i am also holding off on updates until voting is completed, and will be voting on submissions, not edited versions.

Re: Assessment for Judging

Posted: Sat Feb 24, 2018 8:27 am
by M_Tee
I feel bad voting on games I haven't been able to beat.

Is it appropriate to PM requesting savestates for endings or that show essential content a developer might want included in the judging of their game?

Did anyone get stuck in Grunio? If so, I'd gladly PM a password list and a savestate that shows the all-carrots ending.

Re: Assessment for Judging

Posted: Sat Feb 24, 2018 11:49 am
by calima
I only got about two thirds in on Grunio, at that point I just couldn't figure out any solutions. No need to send a password list though.

Re: Assessment for Judging

Posted: Sat Feb 24, 2018 1:40 pm
by na_th_an
There's a cheat in Miedow you can use to skip levels. It's not that hard to find (specially if you check the sources ;) ), but if somebody is interested just PM me.

The ending sequence (which goes from the moment Cheril "awakes" after level 10 to the end and which is playable) is quite nice to experience.

Re: Assessment for Judging

Posted: Sun Feb 25, 2018 1:08 am
by Erockbrox
I am unfamiliar with voting on homebrews or the contest in general. Can anyone vote? When is the voting? Are there rules explaining the voting and contest process?

Re: Assessment for Judging

Posted: Sun Feb 25, 2018 1:58 am
by M_Tee
http://nesdevcompo.nintendoage.com/rules/2017/#judging
Although judging is not open for anyone (essential for preventing it from becoming an internet popularity contest, IMO), I'd still love to hear your assessment of our entry at least.

Re: Assessment for Judging

Posted: Sun Feb 25, 2018 1:15 pm
by na_th_an
I think it would be nice if people posted their own ratings just for fun, specially if commentary was done. I always like to know what people think about my design choices because, as some of you will agree, working for some time in your project basicly blindfolds you.

Re: Assessment for Judging

Posted: Thu Mar 01, 2018 11:03 am
by M_Tee
Thanks for the insight into your approaches.

Judging was definitely enjoyable. It's nice to be motivated to take the time to play and analyze these games more than I may have gotten around to otherwise.

Łukasz and I assessed separately and then planned to submit the average of our scores. I was caught a bit off guard by being restricted to whole number scores. At first, I'd just planned to round up any category entry that fell between whole numbers. But then I realized some games were getting total score boosts of 1.5 points or more (as they had multiple categories being rounded up). So, then we decided to round each (non whole number) category up or down to ensure that no game received more than a half point advantage from rounding.

As for my half of our judging, I assessed each game independently of each other, using the attached sheet which broke down most categories into smaller subcategories and a scale for each.
NesDevMTeeScorecard.pdf
(23.08 KiB) Downloaded 432 times
Anyway, I printed these and penciled them in fairly messily during my last play session, but if anyone wants my individual assessment of their entry, I'll gladly type up a legible copy with notes when I get a chance and shoot it your way.