CAREER: Direct Manipulation Programming Systems
National Science Foundation, Award 1651794 (2017 – 2022)

Computer programming is an increasingly vital and powerful tool, but two realities inhibit even greater impact. First, programming consists of an "Edit-Run-View" workflow that slows the pace of the creative process; only in latter stages can the results be visualized, at which point the programmer must return to text-editing to make subsequent changes. Furthermore, programming is limited to experts. Ideally, users would be aided in the programming process by interactive graphical user interfaces (GUIs), akin to those available for domains such as word processing, spreadsheets, and graphic design.

The goal of this research is to develop theoretical and practical foundations to integrate the expressive power of programming with the ease-of-use of direct manipulation GUIs. The intellectual merits of this research are to answer several open questions about computer programming: (1) How can interactions with the output of a program be used to determine the user's intended modifications to the program? (2) How can interactions with the source code of a program be used to determine the user's intended, higher-level modifications to the program? (3) How can user interfaces integrate the expressiveness of general-purpose programming languages with the intuitive workflow of direct manipulation systems? The project's broader significance and importance are to incorporate answers to these questions into innovative software technology that will (a) be released freely to the public for use by expert and non-expert users, (b) support university-level curricular activities that help bridge computer science with other disciplines such as visual arts, and (c) support middle- and high-school curricular activities that help promote interest in programming and computational thinking.

The project blends and advances techniques in program synthesis, semi-automated refactoring, and human-computer interaction. Three primary activities are pursued: (1) Develop program synthesis algorithms that monitor the previous execution of a program and allow users to specify program modifications by directly manipulating program output. (2) Develop program synthesis algorithms that allow users to specify program modifications by visually and structurally manipulating program text. (3) Design semi-automated programming systems that enable expert and non-expert users to create a variety of digital objects with less text-based editing than in existing programming languages and less mouse-based editing than in existing direct manipulation systems.


  1. Brian Hempel, Justin Lubin, Grace Lu, and Ravi Chugh. Deuce: A Lightweight User Interface for Structured Editing. In Proceedings of the International Conference on Software Engineering (ICSE), Gothenburg, Sweden, May 2018.