Doing MIT's 6.824 Distributed Systems with NUS SoC folks

Mon Feb 01 2021

Introduction

Thanks to Wei Neng for introducing me:

I would do 6.284 if I were you

There's a class in NUS running now

Wei Neng told me to ask Joel, who then told me to consult Bernard, who managed to get me on board! Very pleased.

I was surprised at how progressive NUS was: they allow students to make their own module (they call it DYOM) and do it for credits. So this module is based on the MIT course 6.824: Distributed Systems. The difference between the NUS DYOM course and the MIT course is primarily that of scope. The original 6.824 is a graduate-level course, this is for undergraduates; the original course is taught by a professor, this is student-run; finally, MIT students usually do two modules a semester, while NUS students do five. So I think the plan is to do fewer labs, read fewer papers, and have fewer lectures.

Why am I doing this?

Can think of it as a forcing function: have stopped CS107E because nobody else is doing it with me (Wei Neng lost interest) and it's very easy to blow it off, especially with the inertia of not having UART working with WSL2.

Structure of the course

Read papers, attend lecture/discussion group, and get programming experience implementing the greatest hits. Meeting every Monday in person at NUS SoC.

Papers covered in the course

  1. MapReduce (2004)
  2. GFS (2003)
  3. Fault-Tolerant Virtual Machines (2010)
  4. Extended Raft paper (2014)
  5. Zookeeper (2010)
  6. CRAQ (2009)
  7. Aurora (2017) (replaced with Pacifica (2008) in the 2021 offering)
  8. Frangipani (1997)
  9. Saltzer and Kaashoek (2009). Principles of Computer System Design: An Introduction (used in MIT's 6.033 course). Chapter 9, sections 9.1.5, 9.1.6, 9.5.2, 9.5.3, 9.6.3
  10. Spanner (2012)
  11. FARM (2015)
  12. Spark (2012)
  13. Memcached at Facebook (2013)
  14. COPS (2011)
  15. Bitcoin (2008)
  16. Blockstack (2017)
  17. Analogic FS experience paper

The NUS students are going to vote on the papers they want to discuss, because 17 papers is far too many. There's also another paper, Spinnaker (2011), that was in the 2018 offering of 6.824 which the NUS students want to go through.

I will be writing notes on each paper I read to serve as a reference for my future self, to keep me accountable, and to serve as a proof-of-work.

Lab assignments in the course

  1. Lab 1 implements MapReduce. I've successfully implemented it (took me about a day and a half to read and implement: 7--8 hours total?) My notes on Lab 1.
  2. Lab 2 (A, B, C) implements the Raft consensus algorithm with log compaction. My notes on Lab 2.
  3. Lab 3 (A, B) builds a fault-tolerant key-value store using the Raft consensus mechanism.
  4. Lab 4 is either building a sharded key-value service, or creating a final project of your own. I think this is super cool, and I hope that someone in the class will be keen to do a final project with me.

Other things I expect to learn in the course

  • Programming in Go
  • The Go memory model
  • Concurrency, threads, locking, etc.