Node.js - Streaming 101

Mar 5, 2025

Ever wondered how Netflix loads your favorite show without making you wait for the entire episode to download? Or how Spotify starts playing music instantly? The magic behind all this is streaming, and in the world of Node.js, it’s a game-changer.

🚀 What is Streaming?

Streaming is all about handling data in chunks instead of waiting for the entire thing to load. Think of it like eating a pizza—do you wait for the entire pizza to be ready before taking a bite? Nope. You eat slice by slice. That’s how streams work in Node.js.

🔥 Why Should You Care?

Node.js is built on non-blocking I/O, meaning it can handle huge amounts of data efficiently. Without streams, you'd have to load everything into memory before processing it, which is super inefficient.

Imagine This:

  • You're processing a 5GB file.
  • Without streams: You load all 5GB into memory. 💥 RIP your server.
  • With streams: You process the file chunk by chunk, keeping memory usage low. 🚀

💡 Types of Streams in Node.js

Node.js has four types of streams:

  1. Readable Streams – You read data from a source (e.g., reading a file).
  2. Writable Streams – You write data to a destination (e.g., writing to a file).
  3. Duplex Streams – You can both read and write (e.g., TCP sockets).
  4. Transform Streams – Modify data as it passes through (e.g., compression, encryption).

🛠 Let's Code: Reading a File Stream

Here’s a simple example of reading a file using a Readable Stream in Node.js with ES6 modules:

import { createReadStream } from "fs";

const readStream = createReadStream("bigfile.txt", "utf8");

readStream.on("data", (chunk) => {
  console.log("Received chunk:", chunk.length);
});

readStream.on("end", () => {
  console.log("Finished reading file.");
});

readStream.on("error", (err) => {
  console.error("Error reading file:", err);
});

What’s Happening?

  • createReadStream() creates a stream to read the file in chunks.
  • .on("data") fires every time a chunk is read.
  • .on("end") runs when we’ve finished reading the file.
  • .on("error") handles any errors (because things can go wrong).

✨ Piping Streams Like a Pro

Instead of manually reading and writing data, you can use pipes to transfer data directly.

import { createReadStream, createWriteStream } from "fs";

const readStream = createReadStream("bigfile.txt");
const writeStream = createWriteStream("copy.txt");

readStream.pipe(writeStream);

writeStream.on("finish", () => {
  console.log("File copied successfully!");
});

Why is .pipe() Awesome?

  • No need to manually handle chunks.
  • Handles backpressure (prevents flooding memory if data is too fast).
  • Cleaner, more efficient code.

🔗 Introducing stream.pipeline()

In Node.js 10 (2018), stream.pipeline() was introduced as an alternative to .pipe(), providing better error handling.

Why use pipeline()?

  • Handles errors automatically (no need for multiple .on("error") handlers).
  • Works with async/await.
  • Cleaner and more readable.

How to Use pipeline()

Let’s modify our previous example to use pipeline():

import { createReadStream, createWriteStream } from "fs";
import { pipeline } from "stream/promises";

async function copyFile() {
  try {
    await pipeline(
      createReadStream("bigfile.txt"),
      createWriteStream("copy.txt")
    );
    console.log("File copied successfully!");
  } catch (err) {
    console.error("Pipeline failed:", err);
  }
}

copyFile();

What's Happening?

  • pipeline() connects multiple streams sequentially.
  • If an error occurs, it automatically cleans up the streams.
  • Supports async/await, making it easier to handle in modern Node.js apps.

🎤 Real-World Example: Streaming an HTTP Response Using pipeline()

Let’s say we’re building a Node.js server that streams a file to a user instead of sending it all at once. This time, we’ll use pipeline() to make it cleaner and handle errors automatically.

import http from "http";
import { createReadStream } from "fs";
import { pipeline } from "stream/promises";

const server = http.createServer(async (req, res) => {
  try {
    res.writeHead(200, { "Content-Type": "text/plain" });
    await pipeline(createReadStream("bigfile.txt"), res);
  } catch (err) {
    console.error("Streaming failed:", err);
    res.writeHead(500, { "Content-Type": "text/plain" });
    res.end("Internal Server Error");
  }
});

server.listen(3000, () => {
  console.log("Server running on http://localhost:3000");
});

What's Happening?

  • When a user visits http://localhost:3000, we stream bigfile.txt directly to their browser using pipeline().
  • If anything goes wrong, we handle it with a try/catch and send a proper error response.
  • No manual error handling needed for the stream—pipeline() takes care of it.

🎯 Final Thoughts

Streams in Node.js are powerful and efficient, making them perfect for handling large amounts of data. Whether you're dealing with file processing, HTTP requests, or real-time video/audio, learning streams will level up your Node.js game.

And don’t forget—pipeline() is your friend when handling multiple streams! 🚀

diogoaoliveira.com