[asio] What happens when socket receive buffer overflows?

[Matheus Araújo Aguiar]

Hello List,

I have a network application written with asio that uses TCP. The message
exchange between client and server is done using a protocol that puts a
prefix on each message indicating the number its bytes number. This way, at
one side a prefix is read, and the number of bytes indicated in the prefix
is read as a message, then it reads another prefix and so on.

While testing the application I was having some errors related to reading a
prefix that would indicate a strange number of bytes incoming, a number that
was far greater than the max size possible of message in my protocol. After
assuring that the bytes on the stream were not being merged, i started
suspecting about socket buffer size. So i reduced the socket buffer size to
1K, and the application crashed with a lot less network load than before.

At the moment, i am almost sure that the problem is the socket receive
buffer which overflows and the bytes are discarded, creating a "gap" in the
byte stream which causes an inconsistent state on the application. Does that
make sense? What should happen when the socket receive buffer overflows? It
silently discards the bytes or what?

Thanx for your attention,

[Cliff Green]

Not sure if anyone replied yet or not ...

... protocol that puts a prefix on each message indicating the number its bytes number.

Yes, a fairly common design (I've done a variation on it at least a dozen times).

... errors related to reading a prefix that would indicate a strange number of bytes incoming, a number that was far greater than the max size possible of message ...

Also pretty common problem. I'd bet a major body part there's a bug in the sending (or receiving) side of the app. Something has dropped some bytes somewhere, or lost track of where it is in the "message framing".

... So i reduced the socket buffer size to 1K, and the application crashed with a lot less network load than before. ... At the moment, i am almost sure that the problem is the socket receive buffer which overflows and the bytes are discarded, creating a "gap" in the byte stream which causes an inconsistent state on the application.

As mentioned, you're probably on the mark as to why you're crashing. But not as to the underlying cause.

You're focusing on an OS or TCP stack problem, rather than your code. The TCP protocol has "flow control" built in, and if the receiving side fills up or is congested, it will tell the sending side to quit sending. If packets get dropped while the flow control is "kicking in", the packets will (eventually) get re-transmitted. (On a side note, this type of congestion can be a system-level problem, with certain use cases.)

... What should happen when the socket receive buffer overflows? It silently discards the bytes or what?

TCP never "silently discards" bytes (UDP will, obviously). There's a bug in your code (including the way you're using Asio).

Asio streambufs or iostreams can leave characters in the buffer, and your logic must account for these.

If you're still having the problem, reduce your code to the smallest example that will reproduce the crash, and post it to the mailing list (there's also an Asio mailing list you can use).

Cliff

[Matheus Araújo Aguiar]

Thanks for your reply Cliff,

Post by Cliff Green
Not sure if anyone replied yet or not ...

... protocol that puts a prefix on each message indicating the number

its bytes number.
Yes, a fairly common design (I've done a variation on it at least a dozen times).

... errors related to reading a prefix that would indicate a strange

number of bytes incoming, a number that was far greater than the max size
possible of message ...
Also pretty common problem. I'd bet a major body part there's a bug in the
sending (or receiving) side of the app. Something has dropped some bytes
somewhere, or lost track of where it is in the "message framing".

... So i reduced the socket buffer size to 1K, and the application

crashed with a lot less network load than before. ... At the moment, i am
almost sure that the problem is the socket receive buffer which overflows
and the bytes are discarded, creating a "gap" in the byte stream which
causes an inconsistent state on the application.
As mentioned, you're probably on the mark as to why you're crashing. But
not as to the underlying cause.
You're focusing on an OS or TCP stack problem, rather than your code. The
TCP protocol has "flow control" built in, and if the receiving side fills up
or is congested, it will tell the sending side to quit sending. If packets
get dropped while the flow control is "kicking in", the packets will
(eventually) get re-transmitted. (On a side note, this type of congestion
can be a system-level problem, with certain use cases.)

Probably you are right. Believing that the problem was with the buffer, i
modified the way i read from the socket, in order to make it faster, and the
problem is not occurring anymore.

Post by Cliff Green

... What should happen when the socket receive buffer overflows? It

silently discards the bytes or what?
TCP never "silently discards" bytes (UDP will, obviously). There's a bug in
your code (including the way you're using Asio).
Asio streambufs or iostreams can leave characters in the buffer, and your
logic must account for these.
If you're still having the problem, reduce your code to the smallest
example that will reproduce the crash, and post it to the mailing list
(there's also an Asio mailing list you can use).
Cliff
_______________________________________________
Boost-users mailing list
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Jonathan Franklin]

Post by Matheus Araújo Aguiar
While testing the application I was having some errors related to reading a
prefix that would indicate a strange number of bytes incoming, a number that
was far greater than the max size possible of message in my protocol.

It sounds like you have made the problem go away.

FWIW, on the send side of a TCP connection, you may do 4 individual
writes of 64 bytes each, but on the receive side you may only see 256
bytes of data available on the socket. TCP is a stream protocol, so
at the application level, there is no inherent concept of packets or
messages. TCP *uses* segments, which are packets on "the wire", but
gives you a stream at the application level when you read data from
the socket. So there is no correlation between the number of writes
on the send side, and the number of reads on the receive side. It
only guarantees that you will receive the bytes in the same order they
were sent... If you receive them at all. ;-)

You must design you socket reader with this in mind. Good luck! :-)

Jon

[Matheus Araújo Aguiar]

On Fri, Apr 9, 2010 at 5:52 PM, Jonathan Franklin <

Post by Jonathan Franklin

Post by Matheus Araújo Aguiar
While testing the application I was having some errors related to reading

a

Post by Matheus Araújo Aguiar
prefix that would indicate a strange number of bytes incoming, a number

that

Post by Matheus Araújo Aguiar
was far greater than the max size possible of message in my protocol.

It sounds like you have made the problem go away.
Apparently, yes!
FWIW, on the send side of a TCP connection, you may do 4 individual
writes of 64 bytes each, but on the receive side you may only see 256
bytes of data available on the socket.

TCP is a stream protocol, so

Post by Jonathan Franklin
at the application level, there is no inherent concept of packets or
messages. TCP *uses* segments, which are packets on "the wire", but
gives you a stream at the application level when you read data from
the socket. So there is no correlation between the number of writes
on the send side, and the number of reads on the receive side. It
only guarantees that you will receive the bytes in the same order they
were sent... If you receive them at all. ;-)
You must design you socket reader with this in mind. Good luck! :-)

Thanks for the tip. Actually i was already thinking like that. When i refer
to messages in my post, i want to mean "number of bytes expected before
reading another size of bytes expected" ..

Post by Jonathan Franklin
Jon
_______________________________________________
Boost-users mailing list
http://lists.boost.org/mailman/listinfo.cgi/boost-users

best regards,