GoKawiilApple researchers have published details of SQUIRE, an experimental AI-powered tool that helped developers explore and refine interface ideas with more control than with typical AI coding tools. Here are the details.
An interesting approach to AI-powered interface prototyping
In a new paper titled SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations, a group of Apple developers proposes a novel way to approach AI-generated interfaces.
As they explain, natural language has been bringing more flexibility to multiple aspects of the development process, but it also brings two key challenges:
First, natural language on its own can be ambiguous, making it difficult for developers to precisely communicate their intentions. Second, the model may respond unpredictably, requiring the developer to re-prompt through trial-and-error to repair any undesired changes.
That’s where SQUIRE comes in. It is a visual interface that lets developers build and refine UI prototypes step by step, with clearer control over the results.
From the study:
In SQUIRE, users start a project by providing a prompt that describes their goals for the UI, along with sample data containing information for SQUIRE to use as a reference. Users then construct UI as a tree of components in top-down fashion by prompting SQUIRE to fill holes representing missing yet expected functionality. In response to this kind of request, SQUIRE generates a list of appropriate alternatives, each scoped specifically to the targeted hole in the unfinished UI. Clicking on each alternative immediately updates a live rendered preview as well as underlying code, making it easy to visualize the differences. The user can also pose targeted requests to modify the appearance of specific areas of the UI, with the guarantee that no code outside the intended scope will be mutated. In response to this kind of request, SQUIRE generates ephemeral controls that allow the user to apply semantically-related changes quickly and without re-prompting. In all cases, the LLM acts as a companion, presenting reasonable choices for the user to evaluate, but leaving the user with agency to accept or reject its suggestions.
In other words, natural language prompts were still how developers interacted with SQUIRE, but instead of affecting the entire interface at once, each prompt is tied to a specific part of the UI.
Based on observations with 11 frontend developers who used SQUIRE to develop interface prototypes, the researchers found that participants were able to explore and iterate on different UI designs with a strong sense of control, while also rating the system highly for usability and overall satisfaction.
... continue reading