New York Tech Journal
Tech news from the Big Apple

Beaker Notebook: the #UX of Iterative Data Exploration

Posted on August 12th, 2015

UX + Data

08/12/2015 @ Pivotal Labs, 625 6th Ave, NY

20150812_192833[1] 20150812_193405[1] 20150812_193750[1] 20150812_193942[1]

Jeff Hendy and Scott Draves @TwoSigma presented @Beaker, a lab #notebook on the #web. The notebook allows researchers to collect data, code, graphs & tables for analyses done using one or more programming languages. The tool provides a seamless method to transfer data across a variety of languages making it easy to use the tools from each. Languages currently include: #Python, #R, #Julia, #JavaScript, #Scala, #Ruby, #Node.js, #D3, #Latex, #HTML, …

Beaker was developed by Two Sigma, an investment manager, to give their researcher a tool to analyze markets and document their findings. It is now an open source product.

The notebook is divided into sections and sections can be grouped hierarchically into larger sections. Within a section, an analysis can be performed in Python, for instance, and the output is saved to Beaker variables. These variables can be analyzed using R, Python or any of the supported languages. Beaker can also produce interactive graphics using its own native charting package. The notebook with code, data, and graphs can be saved for further analysis.

Jeff and Scott next talked about the design challenges when creating Beaker. These include:

  1. All languages are fully supported.
  2. Open source
  3. Environment independent

To create an expandable library of supported languages they have an intermediate Beaker language with plug-ins to handle each programming language. To insure Beaker can run on different operating systems, on- and off-the cloud, the user interface is text-based with little formatting.

To accommodate the wide range of programming and data analysis experience across users, they developed several interfaces from verbose (shows language employed, etc.) to terse. To help all levels of users, they adapted the web interface to provide key features available on local desktops, but frequently not available in browsers: 1. Menus in the upper margins, 2. Windows that can be repositioned on the desktop, 3. File dialogs.

To give the web app these functions, they used a framework from ‘The Electron’ which is developed in Chromium incorporating the tools from Node.js.

Data and data structures are passed across languages using #JSON. This offers generality, but with some loss of accuracy for floating point numbers. (in the future they plan to pass values using binary files). They are currently working on methods to share notebook sections (and possibly forked versions).

The audience was invited to try out the system at

posted in:  applications, data, data analysis, Programming, UI, UX, UX+Data    / leave comments:   No comments yet