Netty is an asynchronous networking framework for Java. When you build a network server on the JVM, it sits near the bottom of the stack — and a lot of server software is built on top of it.
The issue
When I found it, issue #17141 had been open for less than ten days — a WebSocket bug report with no PR linked to it yet. Nobody seemed to be working on it, so I decided to try fixing it myself.
The bug lives at the point where a WebSocket connection begins.
A WebSocket connection starts as HTTP.
The client sends an upgrade request — "let's switch this connection to WebSocket" — and once the server accepts with a 101 Switching Protocols response, WebSocket frames flow over the same TCP connection from then on.
The 101 is, in effect, the last HTTP message that will ever cross that connection.
This request-and-response exchange is called the handshake.
How it surfaced
The reporter's setup required that no messages reach the client before the handshake finished, so they added a handler that collects outgoing messages in a queue and releases them once the handshake completes — and after that, connections started timing out. The 101 response had gotten stuck in that queue too.
for the queue to drain → the handshake must finish
for the handshake to finish → the 101 response must go out
for the 101 response to go out → the queue must drain
Spelled out:
- The queue waits only for the handshake to finish.
- But the handshake only finishes once the 101 response goes out.
- And that 101 response is sitting in the queue.
None of these can happen first, so the queue never drains — a deadlock, and to the client, a timeout.
So why did the 101 response go through the queue handler in the first place?
Root cause
The cause was in the path the 101 response took.
The Netty pipeline
Netty creates one pipeline per connection (channel). It's a chain of handlers that process the data moving in and out of that connection. The socket — the part that actually touches the network — is attached at only one end of the pipeline; Netty calls the socket end the head and the opposite end the tail. The pipeline in the issue looked like this.
socket (head) ▶ HTTP codec ▶ handshake handler ▶ queue handler ▶ tail (direction of incoming data)
Everything was fine right up until the server went to send the 101. The client's upgrade request comes in through the socket, becomes an HTTP request at the HTTP codec, and stops at the handshake handler. The request goes no further — the queue handler beyond it never gets the message. Request received, it's now the handshake handler's turn to send the 101 back.
There are two ways to send it out; the message reaches the socket either way, but the two start from different places.
channel.writeAndFlush() → starts at the tail:
queue handler ▶ handshake handler ▶ HTTP codec ▶ socket
ctx.writeAndFlush() → starts at the calling handler's position (here, the handshake handler):
HTTP codec ▶ socket (skips the queue handler)
The difference comes down to what the two objects are.
channel refers to the connection as a whole, so it has no notion of position — a message written through the channel always starts at the tail, the very end of the pipeline.
ctx is given to each handler individually, so it knows its own spot — a message written through it starts right there.
The offending code
The handshake handler has a ctx, so sending the 101 with ctx.writeAndFlush() would have done the job.
But in the actual code, the handler doesn't send the 101 itself.
A helper object called the handshaker builds and sends the response, and the handler calls it like this.
final ChannelFuture handshakeFuture = handshaker.handshake(ctx.channel(), req);handshake() takes a channel, so the handler pulls one out with ctx.channel() and passes it in.
The handshaker calls writeAndFlush() on that channel, and as we saw, a message written through a channel starts at the tail.
And the first handler in from the tail was the queue handler — that's how the 101 response got stuck in the queue.
To recap: the upgrade request stopped at the handshake handler, so the queue handler never even saw it — yet its response, the 101, started at the tail and ran straight into the queue handler.
From the queue handler's point of view, a response showed up for a request it had never seen.
And with no way to hand the handshaker a ctx, the 101 response always had to start at the tail.
The fix
Getting as far as "the handshaker needs to be able to use a ctx" wasn't hard; the question was how.
Then I noticed that the handshaker's own close() — the method that closes a WebSocket connection — was already written to accept either a channel or a ctx, and decided to follow the same pattern.
public ChannelFuture close(Channel channel, CloseWebSocketFrame frame) { ... }
public ChannelFuture close(ChannelHandlerContext ctx, CloseWebSocketFrame frame) { ... }
private ChannelFuture close0(ChannelOutboundInvoker invoker, CloseWebSocketFrame frame,
ChannelPromise promise) {
return invoker.writeAndFlush(frame, promise).addListener(ChannelFutureListener.CLOSE);
}Simply changing the behavior of the existing handshake(Channel, ...) would be the smallest edit, but it's public API, and existing callers may rely on the write starting at the tail — that would break backward compatibility.
So, like close, I decided to add a new overload that takes a ctx and switch only the internal call.
The implementation follows close0 as well.
Channel and ChannelHandlerContext both implement an interface called ChannelOutboundInvoker, so the shared implementation, handshake0, just takes one and calls writeAndFlush() on it.
// The first argument, invoker, decides where the write starts; channel is still passed for other internal uses.
// Called via the Channel overload: handshake0(channel, channel, ...) → starts at the tail (existing behavior, unchanged)
// Called via the ctx overload: handshake0(ctx, ctx.channel(), ...) → starts at the handler's position
invoker.writeAndFlush(response);And the actual bug fix is one line, at the handshake handler's call site.
- final ChannelFuture handshakeFuture = handshaker.handshake(ctx.channel(), req);
+ final ChannelFuture handshakeFuture = handshaker.handshake(ctx, req);Wrapping up
I wrote it up and submitted PR #17192; after two review suggestions — both dropping final from the new overloads — were applied, the PR was merged.
The handshake response now starts at the handshake handler's position, not at the tail of the pipeline.
The fix is slated for the next release, 4.2.17.Final, and carries cherry-pick labels, so it should land on the 4.1 and 5.0 lines as well.