Wolk - Accelerating sustainable innovation

In the realm of software applications, the traditional approach has been to present users with a visual interface—a collection of buttons, menus, and input fields through which they interact with a set of predefined functions. This model has served us well, turning complex code into user-friendly experiences. However, as technology evolves, so does the way we interact with our digital tools.

Enter generative AI, a paradigm shift that promises to redefine our interaction with technology. Generative AI refers to systems that can generate new content or responses based on a set of inputs, often using machine learning models like GPT-4. Instead of clicking through menus, users can simply describe what they want in natural language, and the AI generates the output or performs the task at hand.

How Does This Change the User Experience?

Simplicity: With generative AI, the need for navigating through complex interfaces is greatly reduced. Users can express their intent directly, and the AI interprets and executes the commands. Accessibility: This shift can make technology more accessible to those who may find traditional interfaces challenging, such as individuals with disabilities or the elderly. Efficiency: Generative AI can streamline workflows by cutting down the number of steps needed to achieve a result, potentially increasing productivity.

The Role of Generative AI in Application Design

Generative AI can serve as a virtual assistant within applications, guiding users through tasks, generating reports, or even coding based on a description of the desired functionality. It can also personalize experiences by learning user preferences and adapting its responses accordingly.

An Example

Recently I took a motorcycle trip from Amsterdam to Southern Italy. I asked AI to help me generate a route from A to B, with some limitations like only wanting to drive a maximum of 250km a day. I asked what would be nice places to stop and spend a day, or what could be targets for a scenic route.

It quickly came up with a pretty good route, but as my topographic knowledge is pretty poor, I wanted to see this route on a map, and potentially swap out some stops. To do this however I had to perform the tedious task of entering every location into Google Maps. It would be great if AI could do this for me!

AI as a router

Going back to the concept of an application as a collection of functions, we'll essentially use AI as a router that takes a user's chat message as input and outputs the function the application should execute, and with which parameters. There's many ways to implement this, but using Vercel's excellent AI SDK, it could look something like this:

"use server";

import { createStreamableValue } from "ai/rsc";
import { CoreMessage, streamText, tool } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";

export async function continueConversation(messages: CoreMessage[]) {
  const result = await streamText({
    model: openai("gpt-4-turbo"),
    messages,
    tools: {
      route: tool({
        description: "Get the locations of waypoints on a route",
        parameters: z.object({
          waypoints: z
            .array(
              z.object({
                longitude: z.string().describe("Longitude of the waypoint"),
                latitude: z.string().describe("Latitude of the waypoint"),
              })
            )
            .describe("List of waypoints"),
        }),
        execute: async ({ waypoints }) => waypoints,
      }),
    },
  });

  const stream = createStreamableValue(result.textStream);
  return { message: stream.value, toolResults: result.toolResults };
}

The AI model will receive a list of functions as tools, and will decide based on the user's input and the description of those functions whether to execute something, or just respond with a regular chat message.

The below implementation shows an example of how to implement the integration of AI and UI, again using Vercel's AI SDK:

"use client";

import { Message, useChat } from "ai/react";
import React from "react";
import ReactMarkdown from "react-markdown";
import { MapRef, Marker, Map } from "react-map-gl";
import Pin from "./components/Pin";

import "mapbox-gl/dist/mapbox-gl.css";
import { Waypoint } from "./api/chat/route";

const initialViewState = {
  longitude: 5.249209225090425,
  latitude: 49.4617243375219,
  zoom: 3.8907766170599363,
};

export default function Chat() {
  let ref = React.useRef<MapRef>(null);

  const [waypoints, setWaypoints] = React.useState<Waypoint[]>([]);
  const { messages, input, handleInputChange, handleSubmit } = useChat({
    maxToolRoundtrips: 5,

    async onToolCall({ toolCall }) {
      if (toolCall.toolName === "showRoute") {
        let waypoints = (toolCall.args as { waypoints: Waypoint[] }).waypoints;
        setWaypoints(waypoints);

        let minLat = Math.min(...waypoints.map((w) => w.latitude));
        let maxLat = Math.max(...waypoints.map((w) => w.latitude));
        let minLon = Math.min(...waypoints.map((w) => w.longitude));
        let maxLon = Math.max(...waypoints.map((w) => w.longitude));

        ref?.current?.fitBounds(
          [
            [minLon, minLat],
            [maxLon, maxLat],
          ],
          {
            padding: 80,
          }
        );

        return JSON.stringify(waypoints);
      }

      return;
    },
  });

  return (
    <div>
      <div className="grid grid-cols-2 h-screen overflow-hidden">
        <div className="flex flex-col h-screen p-6">
          <div className="overflow-auto flex-1 flex flex-col gap-3 prose prose-p:first-of-type:mt-0">
            {messages
              ?.filter((m) => Boolean(m.content))
              .map((m: Message) => (
                <div key={m.id} className="flex flex-col gap-1">
                  <strong className="capitalize">{`${m.role}: `}</strong>
                  <ReactMarkdown>{m.content}</ReactMarkdown>
                </div>
              ))}
          </div>

          <form onSubmit={handleSubmit}>
            <input
              value={input}
              placeholder="Say something..."
              onChange={handleInputChange}
              className="border border-border rounded px-4 py-3 w-full"
            />
          </form>
        </div>

        <div>
          <Map
            ref={ref}
            initialViewState={initialViewState}
            style={{ width: "50vw", height: "100vh" }}
            attributionControl={false}
            mapStyle="mapbox://styles/mapbox/streets-v9"
            mapboxAccessToken={yourAccessToken}
          >
            {waypoints.map((place, index) => (
              <Marker
                key={`marker-${index}`}
                longitude={place.longitude}
                latitude={place.latitude}
                anchor="bottom"
              >
                <Pin />
              </Marker>
            ))}
          </Map>
          <pre>{JSON.stringify(waypoints, null, 2)}</pre>
        </div>
      </div>
    </div>
  );
}

In short the application enables the user to send messages to the AI backend. The response can be either a chat message, or a tool call. If it's a tool call, we check which one it is and execute it on the client side. This execution could well be done on the backend, but since our tool call involves Map operations, it's easier and cheaper to perform this on the client side.

In our case the tool call will always be to show a bunch of waypoints on the map.

Now what

This is a very simple example of what an integration of AI and UI could look like, but already enables very powerful features! I can ask for my route, and instantly view it on the map. I can ask AI to swap a waypoint, or I can ask for multiple suggestions and show those on the map. Technically I could even ask the AI for things like showing capitals of countries on the map.

Data-driven development

Initial development of an application like this can be way faster than a traditional UI, and only little decisions have to be made. By analyzing your user's messages, common tasks can be distilled and turned into better UI. Even requests for new features could be detected if your AI is asked for tasks it cannot complete.

Want to have a go at it yourself? Check out the code for this example here!