Ideas to make more scalable

Topics: Business Logic Layer
Dec 26, 2009 at 3:58 PM

I'm using to build an aggregator, which will have (of course) a front page and then up to two dozen section pages with 10 - 20 post summaries per page.

Once we are in full production this means between 240 - 480 posts per day being added to the database. I love everything about and have been able to setup section editing, section layout and placement, publishing, etc. 

The one concern I have is the current version has in memory storage for all posts in the blog. This would quickly cause problems because of the sheer numbers of posts.

I'm considering the following changes and would appreciate any comments:

1. In memory storage of full posts.  Limit this to only to posts that are currently displayed on home and section pages; top posts; and recent posts.  Make changes to DBBlogProvider (I'm using sql database) FillPosts() method to restrict the loading to only these posts.

2. In memory storage of limited post data (title, author, date, and other fields to be determined).  Limit to posts, which are not included in the #1 list and which have not been archived. Use this data in conjunction to the data stored in #1 to display lists for posts retrieved by categories or tags.

3. Create configurable setting for archiving posts after a certain time period or where the number of posts exceeds a certain number. Also create a configurable setting for the number of top and recent posts, which would be cached with list #1. Once a post is archived it may be retrieved through direct database access using the Post.Load() method but would no longer be cached.

4. Rewrite SiteMap.cs to use direct database queries to generate sitemap.axd.

5. Remove the site search mechanism and replace with a custom google search or a direct database search.

My guess is that 95% or better of site visitors will be browsing the home and section page so the in memory storage of full posts for these areas makes sense.  Having configurable settings will allow an administrator to reduce or increase the size of the memory footprint. I also guess that there are many unforeseen issues and that this will take some time. 

What do you think? Is this approach feasible? Is it the best way to address this issue?