Site news wiped clean
Posted in Django
on April 10th, 2009 by
Stephen DeGrace
Well, I wiped out my database table for my news application.
The news application is a generalized service that allows any other application to publish stories into a site-wide news feed simply by creating NewsItems. The Latest News "widget" on the right sidebar uses this application, as does the site-wide news feed (public blogs separately provide their own news feeds, but also publish NewsItems into the general feed).
Anyway, out of the blue one day, the site news feed stopped adding stories. I could not figure out what was wrong for a while, partly because I was too busy to properly sit down and think about it, maybe. I watched what happened in the database when I manually added stories via the shell, and saw that they were adding normally, and staying on refreshing my view of the database, but upon any web request they would disappear. I commented out my entire base template, so that absolutely no models were being instantiated, and still on every request the loss of the new NewsItems would happen.
I wasn't cluing in to what was wrong, and instead I was blaming the database. I thought the table might have become corrupted in some way, because at an earlier phase in the project the gallery application started publishing an unholy amount of stories when galleries were "updated" related to the fact that Django's signalling framework needs (what I view as) an unholy hack to keep it from calling the signalling functions multiple times based on the way it imports your applications when you don't explicitly use the project name in import statements. This is a wart on Django to be sure, but not really relevant to the problem I was having with the news, except that in order to fix the problem caused by multiple calls of functions tied to signals, I had done a lot of manual deletion of news items in the past.
So, I tried exporting the database to back it up, dropping the news_newsitem table, and resyncing it.
And then I realized my real problem. And then I realized I had made some kind of mistake when I dumped my database and the sql file didn't contain any actual data (I guess I'm not used to Postgresql). So, my old news items are gone. But at least I was able to confirm the real problem using my test set at home, which had appeared to be functioning normally in earlier tests but eventually ran into the same news wall as the live site.
One feature of the news app is a pruning middleware that allows you to set a maximum number of news items to keep and when the maximum is exceeded, it deletes the extra. Since it is a middleware, this checking and deleting is tied to web requests, which would explain how the table was getting pruned down to a certain number of stories (25, actually) and how the pruning was happening only tied to web requests. This is the original code for the pruning middleware:
from news.models import NewsItem
from django.conf import settings
class SiteNewsPruningMiddleware(object):
"""
If the number of news items has exceeded NEWS_SITENEWS_MAX_ITEMS, delete
excess items.
"""
def process_request(self, request):
max_items = getattr(settings, 'NEWS_SITENEWS_MAX_ITEMS', 25)
if NewsItem.objects.all().count() > max_items:
items = NewsItem.objects.all().order_by('pub_date')[max_items:]
for item in items:
item.delete()
And this is the corrected code:
from news.models import NewsItem
from django.conf import settings
class SiteNewsPruningMiddleware(object):
"""
If the number of news items has exceeded NEWS_SITENEWS_MAX_ITEMS, delete
excess items.
"""
def process_request(self, request):
max_items = getattr(settings, 'NEWS_SITENEWS_MAX_ITEMS', 25)
if NewsItem.objects.all().count() > max_items:
items = NewsItem.objects.all().order_by('-pub_date')[max_items:]
for item in items:
item.delete()
Oh. My. God. All that for a minus sign (the difference is the .order_by('-pub_date'). The original version of the middleware trimmed the most recent story when the limit was reached (I just use the default limit of 25).
Come to think of it, I think I'm going to set it up so that if you set NEWS_SITENEWS_MAX_ITEMS to -1, it will turn off pruning and save news indefinitely. The way it'swritten, a value of 0 will already result in causing it to empty the table on every request, which in case thatmight some day be a useful property I'm not going to fiddle with.
But anyway, the old news items are gone, but I have no choice but to live with that unless I want to artificially reconstruct them, which I really don't. I'm kind of bummed that it happened over something so stupid, but especially since no one reads this site or subscribes to the news feeds, it doesn't really matter in the end anyway 
Comments:
There is 1 comment on this item.
Post a Comment
* Required field, your email will not be posted.
On April 12th, 2009 mathieu boudreau wrote:
I read your blogs sweetie and they are really good :)