Dremel, the engine behind google's bigQuery

dremel: google's interactive analysis system (big data search engine)

how i got here

• presto query engine -uses-> parquet files (which is an open source implementation of column data structure) -inspired by-> dremel

the urls to read to get familiar with it:

• https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36632.pdf
• http://www.goldsborough.me/distributed-systems/2019/05/18/21-09-00-a_look_at_dremel/
• https://blog.twitter.com/engineering/en_us/a/2013/dremel-made-simple-with-parquet.html
• https://github.com/julienledem/redelm/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper

repetition and definition levels

https://medium.com/@pallerlaakshay/wrapping-head-around-repetition-and-definition-levels-in-dremel-powering-bigquery-c1a33c9695da

parquet:

• intro: https://parquet.apache.org/documentation/latest/
• blogpost: https://blog.twitter.com/engineering/en_us/a/2013/dremel-made-simple-with-parquet.html

dremel:

• algorithm explanation: https://github.com/julienledem/redelm/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper
• paper: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36632.pdf
• blogpost about paper: http://www.goldsborough.me/distributed-systems/2019/05/18/21-09-00-a_look_at_dremel/