Mubin Salehin

Streaming Real-Time OpenAI Data in Next.js: A Practical Guide

Streaming real-time data in iterative chunks rather than waiting for the entire response leads to a way smoother user experience.

Mubin Salehin's photo
Mubin Salehin
·

7 min read

Streaming Real-Time OpenAI Data in Next.js: A Practical Guide

Table of contents

In this tutorial, we'll guide you through two approaches to stream real-time data from OpenAI's API into your Next.js 13 application. In the first approach, we'll demonstrate how to use Node.js native ReadableStream to achieve this. In the second approach, we'll introduce two helpful libraries, ai and openai-edge, to streamline the process. Additionally, we'll explore the benefits of running the API endpoint on Vercel's Edge runtime for optimal performance.

Access the complete source code for both approaches:

Node.js Native Approach

Application Structure

Here's the basic structure of our Next.js application:

- app
  - api
    - chat
      - route.ts
  - page.tsx
- utils
  - OpenAIStream.ts

Setting Up the API Endpoint

Next, we'll create an API endpoint that the client can use to make a POST request. Inside the app/api/chat folder, create a route.ts file with the following content:

// app/api/chat/route.ts (native approach)
import { OpenAIStream } from '@/utils/OpenAIStream';

export async function POST(req: Request) {
  const { prompt } = await req.json();
  const stream = await OpenAIStream({ prompt });
  return new Response(stream, {
    headers: new Headers({
      'Cache-Control': 'no-cache',
    }),
  });
}

This is the API endpoint responsible for streaming data from OpenAI to our Next.js application. Here's a step-by-step description of what this file does:

  1. HTTP POST Request Handling: When a client makes an HTTP POST request to this API endpoint, it expects a JSON payload containing a prompt. The endpoint extracts this prompt from the request.

  2. OpenAI Streaming Request: The extracted prompt is used to create a streaming request to OpenAI's API. It specifies the model, enables streaming, and provides the user's message.

  3. Streaming Data Transformation: The stream received from OpenAI is transformed into a friendly format by utilizing the OpenAIStream utility we will create later.

  4. Streaming Response: Finally, it responds to the client with the transformed stream.

Creating the OpenAIStream Utility

To enable the native approach for streaming real-time data from OpenAI's API, we will create the OpenAIStream utility as follows:

// utils/OpenAIStream.ts (native approach)
import OpenAI from 'openai';
import { createParser, ParsedEvent, ReconnectInterval } from 'eventsource-parser';

interface OpenAIStreamPayload {
  prompt: string;
}

// Create an OpenAI API client
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function OpenAIStream(payload: OpenAIStreamPayload) {
  const encoder = new TextEncoder();
  const decoder = new TextDecoder();

  let counter = 0;

  // Ask OpenAI for a streaming completion given the prompt
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${OPENAI_API_KEY}`,
    },
    method: 'POST',
    body: JSON.stringify({
      model: 'gpt-3.5-turbo',
      messages: [
        {
          role: 'user',
          content: payload.prompt,
        },
      ],
      stream: true, // must enable to get streaming response
    }),
  });

  const stream = new ReadableStream({
    async start(controller) {
      // callback
      function onParse(event: ParsedEvent | ReconnectInterval) {
        if (event.type === 'event') {
          const data = event.data;
          if (data === '[DONE]') {
            controller.close();
            return;
          }
          try {
            const json = JSON.parse(data);
            const text = json.choices[0].delta?.content || '';
            if (counter < 2 && (text.match(/\n/) || []).length) {
              // this is a prefix character (i.e., "\n\n"), do nothing
              return;
            }
            const queue = encoder.encode(text);
            controller.enqueue(queue);
            counter++;
          } catch (e) {
            controller.error(e);
          }
        }
      }

      // stream response (SSE) from OpenAI may be fragmented into multiple chunks
      const parser = createParser(onParse);
      for await (const chunk of response.body as any) {
        parser.feed(decoder.decode(chunk));
      }
    },
  });

  return stream;
}

The OpenAIStream.ts utility is responsible for creating a readable stream that efficiently handles the streamed data, and transforms it for easy consumption by the front-end. We're using the eventsource-parser library to properly parse the Server-Sent Events (SSE) stream response from OpenAI.

Front-End Implementation

Now, let's capture this streaming capability in our Next.js front-end. Inside the app/page.tsx file, we will add the following code:

// app/page.tsx (native approach)
'use client';

import { useState } from 'react';

export default function Home() {
  const [prompt, setPrompt] = useState('');
  const [response, setResponse] = useState('');

  const onTextChange = (e: React.ChangeEvent<HTMLTextAreaElement>) => {
    setPrompt(e.target.value);
  };

  const onSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault();
    await streamResponse();
  };

  const streamResponse = async () => {
    setResponse('');
    const res = await fetch('/api/chat', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        prompt,
      }),
    });

    if (!res.ok) {
      throw new Error(res.statusText);
    }

    // This data is a ReadableStream
    const data = res.body;
    if (!data) {
      return;
    }

    const reader = data.getReader();
    const decoder = new TextDecoder();
    let done = false;

    while (!done) {
      const { value, done: doneReading } = await reader.read();
      done = doneReading;
      const chunkValue = decoder.decode(value);
      setResponse((prev) => prev + chunkValue);
    }
  };

  return (
      <form onSubmit={onSubmit}>
        <label htmlFor="prompt">Ask KafkaBot 😸🤖</label>
        <textarea
          value={prompt}
          onChange={onTextChange}
        />
        <button type="submit">
          Submit
        </button>
      </form>

      {response && (
          {/* Use pre tag to preserve original formatting */}
          <pre>{response}</pre>
      )}
  );
}

In this code, we capture user input, send it to the server via a POST request, and display the streaming response in real-time.

Library-Based Approach

Now, let's explore a more streamlined approach using libraries. We'll make use of two libraries: ai and openai-edge for a simplified setup.

Setting Up the API Endpoint

// app/api/chat/route.ts (library-based approach)
import { Configuration, OpenAIApi } from 'openai-edge';
import { OpenAIStream, StreamingTextResponse } from 'ai';

// Create an edge-friendly OpenAI API client
const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(config);

// Set the runtime to edge for optimized performance
export const runtime = 'edge';

export async function POST(req: Request) {
  const { prompt } = await req.json();

  // Ask OpenAI for a streaming completion given the prompt
  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true, // must enable to get streaming response
    messages: [
      {
        role: 'user',
        content: prompt,
      },
    ],
  });

  // Convert the response into a friendly text-stream
  const stream = OpenAIStream(response);
  // Respond with the stream
  return new StreamingTextResponse(stream);
}

In this approach, we're using the ai library to simplify the OpenAI interaction and the openai-edge library for optimized edge environment support.

Why Edge Functions?

Edge Functions are designed to execute code closer to the end user and have no cold starts (no extra time needed to boot up before executing code), which can result in lower latency and faster response times. They are typically used for tasks like streaming, routing, authentication, and other dynamic operations that can be performed at the edge.

We can set the runtime to 'edge' for optimal performance in one line:

// app/api/chat/route.ts
export const runtime = 'edge';

Front-End Implementation

The front-end implementation for the library-based approach is as follows:

// app/page.tsx (Library-Based Approach)
import { useState } from 'react';
import { useChat } from 'ai/react';

export default function Home() {
  const [prompt, setPrompt] = useState('');
  const [fetching, setFetching] = useState(false);

  const { messages, handleSubmit, handleInputChange } = useChat({
    api: '/api/chat',
    onFinish: () => setFetching(false),
  });

  const lastMessage = messages[messages.length - 1];
  const response =
    lastMessage?.role === 'assistant' ? lastMessage?.content : null;

  const onTextChange = (e: React.ChangeEvent<HTMLTextAreaElement>) => {
    setPrompt(e.target.value);
    handleInputChange(e);
  };

  const onSubmit = (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault();
    setFetching(true);
    handleSubmit(e, {
      options: {
        body: {
          prompt,
        },
      },
    });
  };

  return (
      <form onSubmit={onSubmit}>
        <textarea
          value={prompt}
          onChange={onTextChange}
        />
        <button type="submit" disabled={fetching || !prompt}>
          Submit
        </button>
      </form>

      {response && (
        <div>
          <h3>Response</h3>
          {/* Use pre tag to preserve original formatting */}
          <pre>{response}</pre>
        </div>
      )}
  );
}

This code captures user input (prompt), sends it to the server via a POST request, and displays the streaming response in real-time. It utilizes the ai library for streamlined OpenAI interaction:

  • Use the useChat hook from the package to handle chat interactions. Configure it with the API endpoint (/api/chat) and specify an onFinish callback to update the fetching state.

  • Extract the last message from the messages array from the useChat hook, ensure it's coming from an assistant role, and set the response variable accordingly. This displays the most recent chat response.

Conclusion

In this tutorial, we explored two approaches to stream real-time data from OpenAI's API into a Next.js application. The first approach uses native Node.js ReadableStream, while the second approach leverages the ai and openai-edge libraries for a streamlined setup. Additionally, we highlighted the benefits of running the API endpoint on Vercel's Edge Runtime for optimal performance around real-time data streaming. Choose the approach that best suits your project's requirements and start building dynamic and responsive applications powered by OpenAI!

chatgptchatgptguideNext.jsReactNode.js