Contributing to Netty — Why a WebSocket handshake got stuck waiting for its own response

netty/netty #17141 → PR #17192

I came across a Netty issue about a WebSocket connection timing out. The reporter had tried buffering messages instead of sending them immediately, and the connection stopped completing. mega12345mega, who opened the issue, had already pointed to the code that sent the response from the tail of the pipeline. I followed the code in the issue to trace the outbound path.

All they had added was a queue

Before exchanging WebSocket messages, the client and server need to complete a handshake. The client sends an HTTP request, and the server responds with 101 Switching Protocols. They can then use the same TCP connection for WebSocket messages.

The reporter wanted to queue messages even before the handshake finished. They added a handler that held outgoing messages until it received the HandshakeComplete event. But the queue caught more than WebSocket messages: it also caught the 101 response needed to finish the handshake.

Without that response, the handshake could not complete. Without a completed handshake, the queue would not release the response. Eventually, the WebSocket connection timed out. The queue handler was behind the handshake handler, so why was the 101 response reaching it?

The response started at the tail

Netty processes a connection's data through a sequence of handlers called a pipeline. A simplified version of the arrangement in the issue looks like this. The head is on the left, closest to the socket; the tail is on the right.

head (socket) → HTTP codec → handshake handler → protocol handler → queue handler → tail

The request comes in from the left, passes through the HTTP codec, and is handled by the handshake handler. It does not reach the queue handler behind it. The response, though, starts somewhere else. Calling channel.writeAndFlush() starts at the tail and works back toward the socket, rather than starting where the request was handled. That takes it through the queue handler first.

Writing through a handler's ctx takes a different path. ctx is a ChannelHandlerContext, which knows the handler's position in the pipeline. Calling ctx.writeAndFlush() from the handshake handler starts from that position and heads toward the socket, skipping the queue behind it.

The actual code removes the handshake handler from the pipeline first. Writing through its ctx still starts from the handler's former position. The outbound paths at that point are:

channel.writeAndFlush() → starts at the tail:
  queue handler → protocol handler → HTTP codec → socket

ctx.writeAndFlush() → starts at the removed handshake handler's former position:
  HTTP codec → socket   (skips the queue handler)

The handshake handler therefore needed to send the response through ctx. But it did not create and send the response itself; it delegated that work to WebSocketServerHandshaker. At the time, handshake() accepted only Channel. Even though the handler had a ctx, it had to pass ctx.channel(). The handshaker then wrote through that channel, sending the response back through the queue at the tail.

Letting handshake() accept a context

The handshaker needed a way to accept ctx. Changing the behavior of the existing handshake(Channel, ...) methods would affect external callers, some of which might rely on writes starting at the tail.

The same class already had a useful example in close(). To close a WebSocket connection, callers could pass either a Channel or a ChannelHandlerContext. Both implement ChannelOutboundInvoker, so the internal close0() method accepted that interface and used it to write the message.

public ChannelFuture close(Channel channel, CloseWebSocketFrame frame) { ... }
public ChannelFuture close(ChannelHandlerContext ctx, CloseWebSocketFrame frame) { ... }

private ChannelFuture close0(ChannelOutboundInvoker invoker, CloseWebSocketFrame frame,
        ChannelPromise promise) {
    return invoker.writeAndFlush(frame, promise).addListener(ChannelFutureListener.CLOSE);
}

I followed the same approach for handshake(). I kept the existing methods and added overloads accepting ChannelHandlerContext. Internally, they delegate to handshake0(), which writes the response through a ChannelOutboundInvoker. Callers passing a channel still write from the tail; callers passing a context write from that handler's position.

I changed Netty's internal handler to pass ctx instead of ctx.channel(). The 101 response now starts from where the handshake handler used to be, so it does not enter the queue behind it.

final ChannelFuture handshakeFuture = handshaker.handshake(ctx.channel(), req);

final ChannelFuture handshakeFuture = handshaker.handshake(ctx, req);

Recording writes where the queue had been

For the regression test, I put a handler that records outgoing messages in the position occupied by the queue handler. After the handshake, I checked that no outgoing messages had been recorded. With the old tail-based write, the 101 response would be recorded and the test would fail.

I added tests for both ways of receiving the request: as an aggregated FullHttpRequest and as an unaggregated HttpRequest. Some existing tests also assumed that a handler behind the protocol handler would receive the response. Since the response no longer passed through that position, I changed those tests to retrieve it with the test channel's readOutbound().