OSDC (Open Source Developers’ Conference) 2011 筆記整理

OSDC 於 2010年 3/26, 3/27 舉辦, 在此將會議聽到的做點紀錄.

OSDC 相關資料

• OSDC 2011 的 Schedule 與 投影片 皆於此頁: OSDC 2011
• OSDC 2011 Videos - OSDC 錄影影片
• OSDC IRC Log

OSDC 2011 筆記

Jui-Nan Lin (PIXNET) - HandlerSocket - A NoSQL plugin for MySQL

• 優點
  1. 會寫 binlog, 所以可以做 replication (所以可以做備份)
  2. No duplicate cache
  3. PPT: HandlerSocket - A NoSQL plugin for MySQL
  4. A NoSQL interface for MySQL
  5. libhsclient (app) -> HandlerSocket Plugin (mysqld)
  6. (app)libmysql -> (mysqld) Listener for libmysql -> SQL Layer
  7. 架構圖 HandlerSocket plugin for MySQL
  8. Memcached + MysQL, Cassandra, Apache CouchDB, Amazon SimpleDB
  9. NoSQL 好處: Scalable, Fast
  10. NoSQL 缺點: 不穩定 + Cassandra at Twitter Today - Twitter 放棄 Cassandra
  11. MySQL 非常 stable, HandlerSocket 支援 NoSQL interface for MySQL
  12. HandlerSocket 750,000qps, MySQL 在同樣硬體, 100,000 qps.
      7.5x faster than original MySQL.
  13. Why Handler Socket is fast? => MYSQLparse(void*), my_pthread_fastmutex_lock (這兩個慢)
  14. HandlerSocket 只有對有 index 得條件才能做 Query, 不需要 Open + locking table, 也不用決定要怎麼去用 index, 建立 tmp table, 不需要 unlock table.
  15. HandlerSocket 做了什麼? Plain text-based protocol (可 telnet 進去 debug)
  16. Keep table open for reuse
  17. 只允許 Query 一定要用 index, 但是可以用 limit 等 range query.
      • PHP: php-handlersocket
      • Java: hs4j
      • Python: python-handler-socket
      • Ruby: ruby-handlersocket
      • Perl: Net::HandlerSocket
  18. Read 和 Write port 要分開來開
  19. 設定 my.cnf
      • handlersocket_port = 9998 # for reading
      • handlersocket_port_wr = 9999 # for writing
      • handlersocket_threads = 16 # reader threads (threads 固定), 建議 同 CPU 數
      • handlersocket_threads_wr = 1
• 缺點
  • 不夠安全 (沒有帳號、密碼機制)
  • 寫入時, 不會 update MySQL Query Cache (會讀到舊得資料, 不過據說已經修好)
  • 沒有 INSERT .. ON DUPLICATED KEY UPDATE ... 得功能不 support MySQL auto_increment. (原作者說已經修好了)
• FAQ
• Q: Write 效能如何
  • 於 Ram disk test: 10萬 QPS(Write), 75萬 QPS (Read)
  • 一般 disk test: MySQL 1萬 QPS (Write), 10萬 QPS (Read)
• Q: 同時 Write + Read 沒有 Lock, 不會有問題嗎? 會不會讀到舊得資料?
  • HandlerSocket 下面接得是 Handler Interface, 所以不會讀到舊得資料.
  • InnoDB 會保證是 Transaction 完成後讀到的內容, 不會讀到舊得資料.
• Q: 寫入時, 不會 update MySQL Query Cache (會讀到舊得資料, 不過據說已經修好)
  • (目前, 若使用 SQL Write, 就不會有這個問題.)
• Q: 寫入硬碟得 process 會不會不同
  • HandlerSocket 與 InnoDB 溝通, 所以 InnoDB 負責此部份, HandlerSocket 不管.
• Q: Read port, Write port
  • Maybe 寫 server 程式比較方便 (使用 epoll), read / write thread 是分開得.
• Q: openIndex 第一個參數
  • 1 => resource id, 在此 session 要用此 index.
• Q: 支援 transation
  • no.
• Q: 資料存放在哪邊?
  • 資料是透過 Handler Interface 做存取, 所以後面不管格式是如何, 要看 Table 是用何種格式. (InnoDB, MyISAM)
• Q: 備份
  • 可以使用 mysQLdump, snapshop, mktable sync, mytable checksum 都可以用.

Benjamin Reed (Yahoo!) - Apache ZooKeeper: Taming Distributed Systems

• Research
• Research Labas Around the Globe
• FLP, CAP
• Cloud Computing
  • Elastic - size changes dynamically
  • Scale - large number of services
• Fallacies 錯誤觀念
  • The network is relable.
  • Latency is zero.
  • Bandwidth is infinite.
• 一般常見的分散式系統 (Classic Distributed System)
  • 1 Master -> n Slave
• 可容忍錯誤的分散式系統 (Fault Tolerant Distributed System)
• (Master(1A) + Master(1B)) => Coordination Service, Master (1B) -> nSlave
• 可容忍錯誤的分散式系統 (Fault Tolerant Distributed System)
• (Master(1A) + Master(1B)) => Coordination Service, Coordination Service -> n Slave
• Coordination Service -> n Worker
• What is coordination?
  • Group membership
  • Leader election
  • Dynamic Configuration
  • Status monitoring
  • Queuing
  • Barriers
  • Critical sections
• Serializable vs Linearizability
  • Linearizable writes
  • Serializable read
  • Client FIFO ordering
• Order + wait-free + change events = coordination
• http://hadoop.apache.org/zookeeper/
• ZooKeeper API
• create, delete, setData, getData, exists, getChildren, sync(), setACL,getACL
• services -> YaView -> workers - worker1, worker2 (Ephemerals created by Session X)
• services -> YaView -> locks (s-1) (Sequence appended on create)
• **Leader**
• ZooKeeper Servers
  • *Leader* Server -> n server, 1 server -> n cliecnt
• 2007 開始 ZooKeeper project
• 2008 一到 apache as Hadoop subproject
• 2010 成為 Apache Top Level porject - ZooKeeper
• Conclusions
  • Distributed system are alive and well
  • There is a lot of work to do
  • Open Source is a great way to build your infrastructure
  • Open Source is a great way to build your reputation

hychen - Use command line tool in Python

• PPT: Use command line tool in Python
• Github: ucltip
• apt-get install vim -t maverick
• 轉換成
• apt_get.install('vim', t='maverick')

xdite - Rapid Development in Rails

• PPT: Rapid Development in Rails
• Rapid Development with Rails
  • DRY / Agile / Rapid
  • Rails 3 > Django > Rails 2 > Zend > Symfony >>> CakePHP
  • Rapid = 穩定迅速產出正確的產品
  • Rapid Development => Team Development
  • 消除合作成本
  • 消除學習成本
  • 過去的自己 + 現在的自己 = Team
  • Open source plugin + 自己 = Team
• Vesrion control everything
  • Git
  • db migration
  • Rails package management = Gemfile
  • deployment = Capistrano (rollback back each build easily)
• Automake Everything
  • Capistrano auto deployment + Deploy VMS
• Compress -> 用 Ruby 寫 CSS
• CoffeeScript -> 用 Ruby 寫 JavaScript
• Query Reviwer

ihower - Introduction to RSpec

1. ppt + viemo: RSpec 演講投影片和示範影片
2. PPT: Introduction to RSpec
3. Mocks Arent Stubs
4. RSpec Mocks => 假物件 (Stub)
   • * 測試物件之間的行為
   • 物件導向 是 物件和物件之間的互動
   • 1 Target Object -> n Collaborator
   • Collaborator 無法控制回傳值得外部物件
   • 建構正確的回傳質很麻煩
   • 可能很慢, 拖慢測試速度
5. Stub 回傳設定好的回傳值
   1. stub("user"), 用來測試最後的狀態
6. Mock: 如果沒被呼叫到, 就算測試失敗 - *用來測試行為的發生*
7. Partial mocking and stubbing
8. 一般測試流程 vs Mock 測試流程
   • Given -> When -> Then
   • Given -> Excepet -> Then
9. * Classical Testing vs Mockist Testing
   • 你先決定你要完成的範圍, 然後實做這個範圍的所有物件
10. TDD 搭配 Integration Test, 會用比較多的 stub 和 mock
11. rcov 測試涵蓋度 (看還有哪些可以在寫測試)
12. Capybara 模擬瀏覽器行為
13. * 怎樣寫好測試
    • 一個 it 裡面只有一種測試目的
    • 不要測 Private methods (測試 Public method 就可以涵蓋到測試了)
    • (Private method 只在這個 class 跑, 測 Public 就可以了, 只有 Public 會用到 Private)
    • 透過抽象化介面來測試 (Searcher 測試 HTTParty, Search... 統一都叫 Searcher (抽象化))

• 1. Isolation

• 2. 盡量用較穩定的介面來進行測試

Chia-liang Kao - trading with opensource tools, two years later

• AnyMQ : Message system
• Protovs (javascript 畫圖)
• EC2 Monitor

yappo - Groonga, full text search engine

1. Yappo
2. 1997 -> use grep 當 search engine
3. japanese search engine
4. Kakasi => 日文斷詞.
5. KAKASHI 詞素解析
6. MeCab: Yet Another Part-of-speech and Morphological analysis system.
7. n-gram
8. Japan Search engine
   1. Namazu  http://www.namazu.org
   2. HyperEstraier http://fallabs.com/hyperestraier
   3. Rast http://projects.netlab.jp/rast/
   4. Senna http://qwik.jp/senna/ (mysql full text search)
   5. tritonn http://qwik.jp/tritonn/
   6. Lucene is not domestic in japan.
   7. Groonga http://groonga.org (Senna 開發者新作)
   8. groonga daemon (HTTP, memecached protocol, groonga protocol)
      • suitable for embedding
      • geolocation search

gugod - Same but different

• Gugod 銘言
  • 規格一直改, 時程估不對, 報價報太少, 好像沒下班
  • 專案似無涯, 待辦如坑儒, 日日勤食蟲, 勿使時程誤
  • 規格本無樹, 時程亦未艾, 工程無實務, 報酬何處來
  • 個人開發者也是有團隊合作，這個團隊是由過去、現在、未來的自己組成

Yoshinori TAKESAKO (Tokyo/Japan) - takesako: x86 ASCII programming (16bit/32bit/64bit)

• 講者 Blog: wafful.org - Web Security Blog
• Developer Security. http://namazu.org/~takesako/
• string-based DSL + Fast Parser
• ASCII programming - 轉換, 7個字元 對應 msam 加法/減法器
• http://www.youtube.com/watch?v=jES7pkENQ9c
• http://www.namazu.org/~takesako/
• http://d.hatena.ne.jp/masutaro/20080128/1201507016
• http://www.namazu.org/~takesako/ppencode/demo.html

Jonathan Worthington - Inside A Compiler

• Source Code -> Parsing -> Parse Tree (more trees built here) -> Code Generation -> Cod In Target Language
• AST - argument list

Scott Chacon - Contributing with Git : Reducing the frictions of Open Source collaboration with the Git VCS (github)

1. PPT: Contributing with Git
2. Snapshots, not Patchs
3. 比對最後一次修改
4. Commit A
   • c3d README.txt
   • f13 hello.c
   • -> 取 c3d
5. Commit B
   • c3d README.txt
   • 6d4 hello.c
   • -> 取 6d4
6. Commit C
   • c3d README.txt
   • 6d4 hello.c
   • -> 取 6d4
7. git diff --check
8. git add --patch
9. git fetch
   1. git merge origin/master
   2. git push origin master
10. # push branch
    1. git checkout -b iss53 31b8;
    2. git commit;
    3. git push origin iss53
    4. git fetch # origin/iss53
11. # git rebase
    1. git merge master
    2. git rebase master
    3. git diff c2 c3 > 2-3.patch
    4. 1-2.patch
    5. 2-3.patch
    6. (rebase) c2' -> c3' -> c5 (master)
12. # Mail
    1. git rebase origin/master
    2. git checkout my-top-branch
    3. git format-patch -M origin/master
    4. 0001-..
    5. 0002-...
    6. git send-email *.patch
13. # preson repo
    • git push public
14. # 多個 developer 的 repo 統一彙整
    1. git remote add nick git://github.com/nickh/project.git
    2. git remote add jess git://github.com/jessick/project.git
    3. git fetch nick # nick branch
    4. git fetch jess # jess branch
15. #my repo
    1. public/master
    2. nick/master
    3. jess/master
    4. master
    5. git merge nick/master jess/master
    6. git push public
16. # Requesting pull
    1. git remote add upstream git://github.com/fabpot/symfony.git
    2. git fetch upstream
    3. git rebase upstream/master
    4. git push origin my_feature
    5. git checkout -b my_feature
    6. git commit; commit...
    7. git fetch upstream # => Contact the Maintainer
    8. git request-pull
    9. git request-pull upstream/master origin
17. # Accepting a Pull request
18. 2 options
    1. - Add tham as a remote
       1. git remote add scott git://github.com/schacon/symfony.git
       2. git fetch scott
       3. git merge scott/my_feature
    2. - One-time pull
       1. git pull git@github.com:schacon/symfony.git my_feature
19. # Online Pull Requests (in github)
    • resource
    • contribgit.heroku.com
    • git-scm.com
    • gitref.org
    • progit.org (char 5)

Roger Dingledine - Tor and censorship: lessons learned

• https://torproject.org
• https://bridges.torproject.org/

The Path to Pass into PaaS - Jeffhung

• PPT: The Path to Pass into PaaS

Lightning talk

• Let's use Xslate - Fuji, Goro (gfx) gfuji@gmail.com
  • Xslate is template engine for Perl5.
  • template engine
    • Template::Toolkit
    • HTML::Template
    • Mason
    • Text::MicroTemplate
    • Xslate si the best
• Automatic HTML escaping by default
• Safe: Smart Escaping
• Use
  • Perl6-link syntax
  • TT2-like syntax
  • Readable error messages
  • perldoc Text::Xslate
• 2011/7/22~23 - HIT - hitcon 2011
• Next Generation Online Game Cloud Zillians = Mark 小迪克
  • Cassandra 不穩定
• 自由軟體讓你五分鐘上新聞
  • Scala
  • Splurk + SWT
• We Like LunaTerm - Android BBS Terminal
  • PPT: http://ppt.cc/Pcpa
  • https://github.com/albb0920/LunaTerm
  • (按讚 內容自動顯示的 extension)
• $4 http://about.me/fourdollars
  • Debian installer 自動安裝
  • boot: auto url=fd.idv.tw
  • boot: 輸入 "auto url=fd.idv.tw"
  • 有 dhcp 的環境, 40mins 就可以裝完
  • fbterm -i fbterm_ucimf
• Fred - ibus-chewing 新酷音, 單純注音
  • ibus-chewing 單純注音模式 - OSDC.tw 2011 Lightening Talk
  • ibus 注音根本沒正常過
  • 救星: ibus-chewing
  • patch 已經上 Debain upstream
• Coscup 8/20~21
  • pi * z * z * a
  • 1000人, 每分鐘 15.0008人
  • 1000/15 = 66.666 分鐘爆滿
  • 五月徵稿 - Gadgets beyon smartphones
  • 七月報名
  • coscup.org

相關網頁

• 2011 OSDC Day 1 筆記
• 跟大師學簡報技巧