- Author
Gao Peng <funky.gao@gmail.com>
- Description
kaixin dlog monitor framework, simulation of hadoop map reduce framework
- Revision
$Id$
Table Of Contents
This is a single-node map reduce framework implementation. It is widely used in kaixin's dlog analysis and reporting system.
/Users/gaopeng/github/dlogmon/src/github.com/stathat/consistent
/Users/gaopeng/github/dlogmon/src/github.com/msgpack
why first strings.Contains(100.123) will be slower to 280000 per sec
enhance performance of strings.Contains
- signal skipped not work
- chain of workers and reducers
Job abstraction
it has member Option
MapReduce
Input -> Map -> Shuffle -> Reduce -> Output
- select count(*) as count, service from dlog.20121212 where time>2000 group by service order by count desc
- if group, each group is a dedicated reducer and goroutine
func return chan
func f() chan int { ch := make(chan int) go func() { for i:=0; i<10; i++ { ch <- i } }() return ch } x := f() println(<-x)