CAREER: Direct Manipulation Programming Systems
National Science Foundation, Award 1651794 (2017 – 2022)



Computer programming is an increasingly vital and powerful tool, but two realities inhibit even greater impact. First, programming consists of an "Edit-Run-View" workflow that slows the pace of the creative process; only in latter stages can the results be visualized, at which point the programmer must return to text-editing to make subsequent changes. Furthermore, programming is limited to experts. Ideally, users would be aided in the programming process by interactive graphical user interfaces (GUIs), akin to those available for domains such as word processing, spreadsheets, and graphic design.

The goal of this research is to develop theoretical and practical foundations to integrate the expressive power of programming with the ease-of-use of direct manipulation GUIs. The intellectual merits of this research are to answer several open questions about computer programming: (1) How can interactions with the output of a program be used to determine the user's intended modifications to the program? (2) How can interactions with the source code of a program be used to determine the user's intended, higher-level modifications to the program? (3) How can user interfaces integrate the expressiveness of general-purpose programming languages with the intuitive workflow of direct manipulation systems? The project's broader significance and importance are to incorporate answers to these questions into innovative software technology that will (a) be released freely to the public for use by expert and non-expert users, (b) support university-level curricular activities that help bridge computer science with other disciplines such as visual arts, and (c) support middle- and high-school curricular activities that help promote interest in programming and computational thinking.

The project blends and advances techniques in program synthesis, semi-automated refactoring, and human-computer interaction. Three primary activities are pursued: (1) Develop program synthesis algorithms that monitor the previous execution of a program and allow users to specify program modifications by directly manipulating program output. (2) Develop program synthesis algorithms that allow users to specify program modifications by visually and structurally manipulating program text. (3) Design semi-automated programming systems that enable expert and non-expert users to create a variety of digital objects with less text-based editing than in existing programming languages and less mouse-based editing than in existing direct manipulation systems.

Publications

  1. Justin Lubin, Nick Collins, Cyrus Omar, and Ravi Chugh. Program Sketching with Live Bidirectional Evaluation. In Proceedings of the ACM on Programming Languages (PACMPL), Issue ICFP, Jersey City, NJ, August 2020. To appear.

  2. Brian Hempel and Ravi Chugh. Tiny Structure Editors for Low, Low Prices! (Generating GUIs from toString Functions). In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Dunedin, New Zealand, August 2020. To appear.

  3. Brian Hempel, Justin Lubin, and Ravi Chugh. Sketch-n-Sketch: Output-Directed Programming for SVG. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), New Orleans, LA, October 2019.

  4. Justin Lubin and Ravi Chugh. Type-Directed Program Transformations for the Working Functional Programmer. In Proceedings of the Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU), New Orleans, LA, October 2019.

  5. Mikaël Mayer and Ravi Chugh. A Bidirectional Krivine Evaluator. In Proceedings of the International Workshop on Bidirectional Transformations (Bx), Philadelphia, PA, June 2019.

  6. Mikaël Mayer, Viktor Kunčak, and Ravi Chugh. Bidirectional Evaluation with Direct Manipulation. In Proceedings of the ACM on Programming Languages (PACMPL), Issue OOPSLA, Boston, MA, November 2018.

  7. Brian Hempel, Justin Lubin, Grace Lu, and Ravi Chugh. Deuce: A Lightweight User Interface for Structured Editing. In Proceedings of the International Conference on Software Engineering (ICSE), Gothenburg, Sweden, May 2018.
September 2019 May 2019

Education and Outreach

New UChicago Course — CMSC 11111: Creative Coding

This course is an introduction to programming, using exercises in graphic design and digital art to motivate and employ basic tools of computation (such as variables, conditional logic, and procedural abstraction). We will write code in JavaScript or related technologies, and we will work with a variety of digital media, including vector graphics, raster images, animations, and web applications. Throughout the course, we will reflect on how graphical user interfaces of the future might unleash the fundamental building blocks of programming for everyday computer use.

First offering: Spring 2021
Satisfies: Mathematical Sciences requirement
Satisfies: Media Arts and Design minor