Twitter problems

I was reading Quora and I found this post.  http://www.quora.com/Twitter-Inc-company/Why-does-Twitter-need-300-employees  Jane mentioned some really interesting CS problems

I remember my housemate was once incredulous as to why we needed so many engineers for such an apparently simple site. 1) Twitter's not as simple a website as you think--keeping Justin Bieber from dominating the trends and user recommendations are two examples of nontrivial features. 2) Like Facebook, the core functionality is simple and most people could do it in a weekend, but getting it all to work at scale is hard.


Here are a few ideas for how to tackle those problems.

1) Keep Justin Bieber off of trends

For each potential trend, keep track of two pieces of information.  How often it normally occurs, and how often it is currently occurring.  List the item as a trend once its current rate greatly exceeds its average rate.

2) Recommendations

Idea 1: Graphs - This is the most feasible, and least out there

1. For a user U, make a graph of every user U follows (following represents an edge). 
2. For every user in the graph, add all users they follow
3. (optional) repeat 2 until graph is big enough
4. Recommend the nodes in the graph with the most inbound edges. 

Idea 2: Clustering

I want to flush this out later, but the basic idea is to create some N dimensional space.  Each dimension corresponds to level of interest in an area. (eg. interest in Justin Beiber)  Map the user and potential recommendations to this space.  Recommend the nearest items.

Figuring out what are potential recommendations and what the space looks like might be either be hard (computationally) or take some extra thinking.

Idea 3: Neural Networks