FeedBurner Architecture | High Scalability

来源：百度文库 编辑：神马文学网 时间：2024/05/24 02:09:05
High Scalability Building bigger, faster, more reliable websites.
Home
Start Here
Forums
Real Life Architectures
All Weblinks
Glossary
Submit a Link
Contact

User login
Username: *
Password: *
Create new account
Request new password

Navigation
Blogs
Create content
Forums
Polls
Recent posts
Categories
Useful Books
Useful Products
Useful Blogs
Useful Papers
Future Events
Home
FeedBurner Architecture

Fri, 07/13/2007 - 00:34 —Todd Hoff
 
FeedBurner Architecture (6)
FeedBurneris a news feed management provider launched in 2004. FeedBurnerprovides custom RSS feeds and management tools to bloggers, podcasters,and other web-based content publishers. Services provided to publishersinclude traffic analysis and an optional advertising system.
Site:http://www.feedburner.com
 
Information Sources
*FeedBurner - Scalable Web Applications usingMySQL andJava
 
Platform
* Java
* MySQL
* Hibernate
* Spring
* Tomcat
 
What‘s Inside?
* The Stats
- FeedBurner is growing faster than MySpace and Digg with 385% trafficgrowth. Total feeds: 808,707, Number of publishers: 471,686.
- 11 million subscribers in 190 countries
* Scaling History
- July 2004: 300Kbps, 5,600 feeds, 3 app servers, 3 web servers 2 DB servers, Round Robin DNS
- April 2005: 5Mbps, 47,700 feeds, 6 app servers, 6 web servers (same machines)
- September 2005: 20Mbps, 109,200 feeds
- Currently: 115 Mbps, 270,000 feeds, 100 Million hits per day
*Scalability Problem 1: Plain old reliability
- Single-server failure, seen by 1/3 of all users
- Health Check all the way back to the database that is monitored byload balancers to route requests in to live machines on failure.
- UseCacti andNagios for monitoring. Using these tools you can look at uptime and performance to identify performance problems.
* Scalability Problem 2: Stats recording/mgmt
- Every hit is recorded which slows everything down because of table level locks.
- Used Doug Lea’s concurrency library to do updates in multiple threads.
- Only stats for today are calculated in real-time. Other stats are calculate lazily.
*Scalability Problem 3: Primary DB overload
- Use master DB for everything.
- Balance read and read/write load
- Found where we could break up read vs. read/write
- Balanced master vs. slave load
* Scalability Problem 4: Total DB overload
- Everything slowed down, was using the database has cache, used MyISAM
- Add caching layers. RAM on the machines, memcached, and in the database
* Scalability Problem 5: Lazy initialization
- When stats get rolled up on demand popular feeds slowed down the whol system
- Turned to batch processing, doing the rollups once a night.
* Scalability Problem 6: Stats writes, again
- Wrote to the master too much. More data with each feed. Added more stats tracking for ads, items, and circulation.
- Use merge tables. Truncate the data from 2 days ago.
- Went to horizontal partitioning: ad serving, flare serving, circulation.
- Move hottest tables/queries to own clusters.
* Scalability Problem 7:Master DB Failure
- Using a primary and slave there‘s a single point of failure becauseit‘s hard to promote a slave to a master. Went to a multi mastersolution.
* Scalability Problem 8: Power Failure
- Needed a disaster recovery/secondary site.
- Active/active not possible. Too much hardware, didn‘t like havinghalf the hardware going to waste, and needed a really fast connectionbetween data centers.
- Create custom solution to download feeds to remote servers.
 
Lessons Learned
* Know your DB workload, Cacti really helps with this
* ‘EXPLAIN’ all of your queries. Helps keep crushing queries out of the system
* Cache everything that you can
* Profile your code, Usually only needed on hard-to-find leaks
To discuss this article please visit the forums at
http://highscalability.com/feedburner-architecture-0
Example
Java
Memcached
MySQL
Visit FeedBurner Architecture
58 reads
Comments
Post new comment
Your name:
E-mail: The content of this field is kept private and will not be shown publicly.
Homepage:
Subject:
Comment: *
Web page addresses and e-mail addresses turn into links automatically.
Allowed HTML tags: