It is currently Fri Dec 15, 2017 8:40 pm

All times are UTC - 7 hours





Post new topic Reply to topic  [ 10 posts ] 
Author Message
 Post subject: Missing messages?
PostPosted: Sun May 31, 2009 9:05 am 
Offline
User avatar

Joined: Tue Jun 24, 2008 8:38 pm
Posts: 1519
Location: Fukuoka, Japan
It seems the BBS had a few issues during the night and some message disappeared. Is it because of some maintenance or the BBS software crashed and a few mail were lost? A few threads seems affected.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 31, 2009 11:51 am 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10165
Location: Rio de Janeiro - Brazil
I tried posting last night but failed. Some debugging information regarding keywords showed up and the post wasn't made. I haven't noticed anything missing yet.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 31, 2009 12:12 pm 
Offline
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3192
Location: Mountain View, CA, USA
This was the result of somewhat unplanned maintenance on all the servers in our co-location facility. (We had a lot of work to do, and were there for about 4 hours doing numerous upgrades, re-wired the entire cage, and many other things).

There was only one piece of fallout: the MySQL server began acting oddly prior to the maintenance beginning, specifically with regards to NFS. Anyone familiar with UNIX knows how NFS timeouts can more or less indefinitely stall a userland program, and we had many of those. I'm pretty sure I found the root cause and fixed it, but by that time the damage had been done.

We had to reboot the MySQL server without cleanly shutting things down. Specifically, shutdown, reboot, etc. would all cause disk buffers to get flushed -- and that includes NFS -- so we had to tell the kernel to shut down without flushing any I/O buffers (e.g. any cached I/O transactions would be lost) using "reboot -q -n". This is a big no-no in the BSD world, but the circumstances justified it.

Sadly, this had a major effect on MySQL. There were 8 or 9 tables which mysqlcheck reported as corrupted, and using --repair fixed them, but some rows were lost. Thus, there could be some table integrity problems (e.g. two INSERT statements were made in 2 different tables, one went through but the other was lost, so the board software becomes confused).

Two other users saw similar impact -- one lost most of his entire website (but had backups of the Wordpress content so he should be fine), and the other also lost forum posts (also phpBB).

This is the first time we've actually seen data loss of this severity before. There was not a hardware malfunction or anything of that sort -- no disk errors/bad sectors/etc.. This was purely due to the machine becoming more or less deadlocked due NFS problems on the MySQL server.

This shouldn't happen again in the future given the modifications I've made to the NFS configuration on the server. But if there's any brokenness that people find here (besides missing posts), please let me know and I'll see if I can reverse engineer phpBB and delete or fix whatever rows are broken.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 31, 2009 7:21 pm 
Offline
User avatar

Joined: Tue Jun 24, 2008 8:38 pm
Posts: 1519
Location: Fukuoka, Japan
tokumaru wrote:
I tried posting last night but failed. Some debugging information regarding keywords showed up and the post wasn't made. I haven't noticed anything missing yet.


One of your mail on my thread regarding interleaving was deleted, my answer to it too. Some of the message in the gemventure thread, a few in Nescicide new screens. Except for that, I don't remember anything else was deleted.

It's not like it was that much important information so we will survive.


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 31, 2009 8:58 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10165
Location: Rio de Janeiro - Brazil
Yeah, I ended up noticing. Kinda scary to think that we could loose all the content overnight like that! O_O


Top
 Profile  
 
 Post subject:
PostPosted: Sun May 31, 2009 10:22 pm 
Offline
User avatar

Joined: Tue Jun 24, 2008 8:38 pm
Posts: 1519
Location: Fukuoka, Japan
tokumaru wrote:
Yeah, I ended up noticing. Kinda scary to think that we could loose all the content overnight like that! O_O


Since they're doing backup of the database (I think), the worst that could happen is losing one day of message.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 01, 2009 11:13 am 
Offline
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3192
Location: Mountain View, CA, USA
Tom reported some threads were showing as multi-page, even though their posts were missing (as a result of the mishap described above).

A simple MySQL query can get me the list of post IDs which exist in the phpbb_posts table but not in the phpbb_posts_text table:

Code:
mysql> SELECT a.post_id,a.topic_id FROM phpbb_posts AS a LEFT JOIN phpbb_posts_text AS b ON a.post_id = b.post_id WHERE b.post_id IS NULL;
+---------+----------+
| post_id | topic_id |
+---------+----------+
|   47427 |     5230 |
|   47428 |     5229 |
|   47429 |     5217 |
|   47430 |     5232 |
|   47431 |     5217 |
|   47432 |     5217 |
|   47433 |     5230 |
+---------+----------+
7 rows in set (0.00 sec)

And then to determine what the threads were which were affected:

Code:
mysql> SELECT topic_id,topic_title FROM phpbb_topics WHERE topic_id=5230 OR topic_id=5229 OR topic_id=5217 OR topic_id=5232;
+----------+----------------------------------+
| topic_id | topic_title                      |
+----------+----------------------------------+
|     5217 | GemVenture                       |
|     5229 | NESICIDE v? screenshots          |
|     5230 | Map data interleaving, worth it? |
|     5232 | converting problem               |
+----------+----------------------------------+
4 rows in set (0.01 sec)

So there you have it -- those were the 4 threads which lost some posts.

I've since deleted all of the post_ids shown in the above list.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 01, 2009 2:07 pm 
Offline
Site Admin
User avatar

Joined: Mon Sep 20, 2004 6:04 am
Posts: 3488
Location: Indianapolis
Thanks for doing the maintenance and repairs, man. Parodius is like my home, it's the first and last place I go (online) everyday. I know many people appreciate it, but probably fewer say it (or even know where this stuff is hosted). So, I thought I'd say so.

BTW I want to remind people that a possible way to show appreciation is to donate to our hosts - there's a paypal link on http://www.parodius.com/ on the upper right. This site has been on this server for over a decade, without ever - not even once - having paid advertisements. I don't know if that can be said for many other sites.

And surprisingly, as far as counting hits on the mainpage of NESdev goes, I recently noticed it has actually turned around and is trending upwards after years of slow and steady decline (and no HTML updates).


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 01, 2009 6:37 pm 
Offline
User avatar

Joined: Tue Jun 24, 2008 8:38 pm
Posts: 1519
Location: Fukuoka, Japan
This seems to cover the post that I remember going strange and some extra that I didn't remember.

Thanks for fixing the DB.

It's a nice little BBS here. This is the only one that I go everyday.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 01, 2009 6:59 pm 
Offline
Formerly ~J-@D!~
User avatar

Joined: Sun Mar 12, 2006 12:36 am
Posts: 445
Location: Rive nord de Montréal
Banshaku wrote:
This seems to cover the post that I remember going strange and some extra that I didn't remember.

Thanks for fixing the DB.

It's a nice little BBS here. This is the only one that I go everyday.

me too!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC - 7 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group