National Science Foundation, Award 1651794 (2017 – 2022)
Computer programming is an increasingly vital and powerful tool, but two
realities inhibit even greater impact. First, programming consists of an
"Edit-Run-View" workflow that slows the pace of the creative process; only in
latter stages can the results be visualized, at which point the programmer must
return to text-editing to make subsequent changes. Furthermore, programming is
limited to experts. Ideally, users would be aided in the programming process by
interactive graphical user interfaces (GUIs), akin to those available for
domains such as word processing, spreadsheets, and graphic design.
The goal of
this research is to develop theoretical and practical foundations to integrate
the expressive power of programming with the ease-of-use of direct manipulation
GUIs. The intellectual merits of this research are to answer several open
questions about computer programming: (1) How can interactions with the output
of a program be used to determine the user's intended modifications to the
program? (2) How can interactions with the source code of a program be used to
determine the user's intended, higher-level modifications to the program? (3)
How can user interfaces integrate the expressiveness of general-purpose
programming languages with the intuitive workflow of direct manipulation
systems? The project's broader significance and importance are to incorporate
answers to these questions into innovative software technology that will (a) be
released freely to the public for use by expert and non-expert users, (b)
support university-level curricular activities that help bridge computer science
with other disciplines such as visual arts, and (c) support middle- and
high-school curricular activities that help promote interest in programming and
computational thinking.
The project blends and advances techniques in program synthesis, semi-automated
refactoring, and human-computer interaction. Three primary activities are
pursued: (1) Develop program synthesis algorithms that monitor the previous
execution of a program and allow users to specify program modifications by
directly manipulating program output. (2) Develop program synthesis algorithms
that allow users to specify program modifications by visually and structurally
manipulating program text. (3) Design semi-automated programming systems that
enable expert and non-expert users to create a variety of digital objects with
less text-based editing than in existing programming languages and less
mouse-based editing than in existing direct manipulation systems.
