Smörgåsbord

A while ago I received a question from a reader. He was having some trouble with a pretty interesting setup. As I was rather intrigued I spent some time to figure it out — it’s solved, but I’m not 100% sure on the theory. So if you are, please comment.

the question

from Yves <theYinYeti@yeti.selfip.net>
to wicher@gavagai.eu
date Wed, Sep 15, 2010 at 16:06
subject Socat

Dear mister Minnaard,

I hope you don’t mind me asking you for help. By chance, I saw your blog post:
http://smorgasbord.gavagai.nl/2010/01/socat-access-your-x-servers-domain-socket-over-tcp/

It happens I have a problem with socat on my Sheevaplug server, and it seems you have good knowledge of this software.

Following http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, I use socat to tunnel DNS through SSH, so as to have transparent DNS resolution, to be used with a transparent proxy (NAT rules + tinyproxy, backed by cntlmd, backed by corporate proxy).

My problem is that the number of socat PIDs on both the client and the server keeps growing. I’ve had floods of “accept: too many open files” messages, and I suspect socat is the culprit (at this point, a ps -ef shows almost a thousand of socat PIDs!); especially since the transparent DNS stops working after this flood of messages.

More specifically, on the client, I run:

• dnsmasq with “server=127.0.0.1#1053” as a back-end

• socat -ly udp4-listen:1053,reuseaddr,fork tcp:127.0.0.1:1054

• ssh -L 1053:208.67.222.222:53 -L 1054:127.0.0.1:1054 \
  -Tfx remoteuser@remoteserver \
  'socat -lf /dev/null tcp4-listen:1054,reuseaddr,fork \
  UDP:208.67.222.222:53'

  (208.67.222.222 is OpenDNS)

Besides, on localhost (local client), ssh is configured this way:

Host remoteserver
Port 443
ServerAliveInterval 60
ProxyCommand /usr/bin/proxytunnel -v -p corporateproxy:8080 -P login:passwd -d %h:%p

Do you have an idea on how I could make socat behave better?
I thank you beforehand for any advice you could provide.
Sincerely,

Yves.

the reply

from Wicher Minnaard <wicher@gavagai.eu>
to Yves <theYinYeti@yeti.selfip.net>
date Thu, Sep 16, 2010 at 02:19
subject Re: Socat

Dear mister Yves,

On Wed, Sep 15, 2010 at 16:06, Yves <yves@yeti.selfip.net> wrote:

Dear mister Minnaard,

I hope you don’t mind me asking you for help. By chance, I saw your blog post:
http://smorgasbord.gavagai.nl/2010/01/socat-access-your-x-servers-domain-socket-over-tcp/

It happens I have a problem with socat on my Sheevaplug server, and it seems
you have good knowledge of this software.

Thank you for your compliment. Actually, I do not know socat all too well, but I know a thing or two about networking.

Following http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, I use socat to tunnel DNS through SSH, so as to have transparent DNS resolution, to be used with a transparent proxy (NAT rules + tinyproxy, backed by cntlmd, backed by
corporate proxy).

Sounds like just another day at the office ;-)
Interesting setup.

My problem is that the number of socat PIDs on both the client and the server keeps growing. I’ve had floods of “accept: too many open files” messages, and I suspect socat is the culprit (at this point, a ps -ef shows almost a thousand of socat PIDs!); especially since the transparent DNS stops working after this flood of messages.

• dnsmasq with “server=127.0.0.1#1053” as a back-end

• socat -ly udp4-listen:1053,reuseaddr,fork tcp:127.0.0.1:1054

• ssh -L 1053:208.67.222.222:53 -L 1054:127.0.0.1:1054 \
  -Tfx remoteuser@remoteserver \
  'socat -lf /dev/null tcp4-listen:1054,reuseaddr,fork \
  UDP:208.67.222.222:53'

  (208.67.222.222 is OpenDNS)

Sidenote: Google provides a solid DNS service nowadays. I don’t know what they do with the data, though the same could be said about OpenDNS.
See http://code.google.com/speed/public-dns/


Host remoteserver
Port 443
ServerAliveInterval 60
ProxyCommand /usr/bin/proxytunnel -v -p corporateproxy:8080 -P login:passwd -d %h:%p


So it’s DNS over UDP over TCP over SSH over proxied HTTPS, or something. I’m losing track of all the layers here.

Do you have an idea on how I could make socat behave better?
I thank you beforehand for any advice you could provide.

Well you got me curious now.
I managed to reproduce your problem in a simplified setup:

- on host w00tage, setup the nameserver façade:
socat udp4-listen:1053,reuseaddr tcp:localhost:1054

- from host w00tage, tunnel the TCP connection over SSH to the host hosting the to-be-proxied nameserver:
ssh -L 1054:localhost:1054 priv

- now, on this host priv, connect the TCP socket to the UDP socket where the nameserver is listening
socat tcp4-listen:1054,fork,reuseaddr UDP:localhost:53

- on w00tage, do queries on localhost:1054:
dig @localhost -p1053 google.fr

Dig is part of the bind-tools package and greatly helps in diagnosing DNS problems.
The first dig will probably return. Subsequent digs may take seconds to return. And they leave stray socat processes behind on both w00tage and priv. Kill the strays on w00tage, and the ones on priv dissolve. So the local ones (on w00tage) tie up the remote ones. The latter are not the problem, just a symptom. You can keep on querying, doing more digs, until your file descriptors run out.

A stab at what is going on here: UDP is stateless, and socat is ignorant of protocol data (as it should be). A DNS query takes one UDP packet. But socat does not know we’re doing DNS. And since UDP is stateless, there’s no ‘bye’ in the conversation. So socat doesn’t know when to stop waiting for more. Dig does, though – it recognizes a DNS answer when it sees one. So dig returns (but why the delays? socat buffers and timeouts maybe?) but socat doesn’t. It’s still waiting for more.
Now since we’re forking socat for every new independent socketpair opened (which is what happens when you do a query), we get a lot of those hanging socats.

I had a look at the URL you sent me, http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, to see how they are getting it to work and it appears there is a ‘UDP-RECVFROM’ address type in socat. From its manpage:
————
UDP-RECVFROM: Creates a UDP socket on [UDP service] using UDP/IP version 4
or 6 depending on option pf. It receives one packet from an
unspecified peer and may send one or more answer packets to that peer.
This mode is particularly useful with fork option where each arriving
packet – from arbitrary peers – is handled by its own sub process.
This allows a behaviour similar to typical UDP based servers like
ntpd or named. This address works well with socat UDP-SENDTO address
peers.
————

Hey, explicit mentioning of nameservers there. Receive one packet, don’t wait for more, send it off, collect answer, send that back.
On host w00tage, I run the façade thusly:

socat udp4-recvfrom:1053,fork,reuseaddr tcp:localhost:1054

And that appears to work indeed. But wait, how does socat (on priv) know when to stop trying to collect answer packets?
I think it’s socat timeouts, but I’m not 100% sure. What leads me to think that it’s timeouts is that when I stress-test this setup from w00tage:

while true; do dig @localhost -p1053 sheeva; done
(for ’sheeva’ the nameserver never has to do recursion)
I have about 30 forked-off query-listeners when I do a pgrep socat | wc -l on w00tage. But I have about 400 on priv. That number goes up and down and the pids change, so they do not *accumulate* but rather seem to hang about until they decide their job is done. But to be sure I’d have to study the source. Or ask the socat mailing list.

Anyway, problem solved!

I was happy to help you and I got some exercise out of it. Any reason why you didn’t post this question as a comment on my blog? Maybe it would be a bit off-topic, I can see that.
Would you mind if I published this?

Regards and good luck, Wicher Minnaard.

it works

from Yves <theYinYeti@yeti.selfip.net>
to wicher@gavagai.eu
date Thu, Sep 16, 2010 at 10:01
subject Re: Socat

Dear mister Minnaard,

All in all, I was just careless in adapting the “alternative solution with socat” to my setup.

I thank you for your analysis and tests; I learnt a few things following them. And I certainly wouldn’t mind at all if you were to publish this.
However, if you do so, please change the e-mail to “theYinYeti@yeti.selfip.net” (the Sheevaplug :-) ), and my name to simply “Yves”; thank you. As for the web site (I see there’s a field for it), it is “http://yeti.selfip.net/gablin.php” (a bit lame, but I may find time someday to update it…)

As you said, I had the feeling this was a bit off-topic. But now, I see how this can shed some light on advanced socat usage.
–
Regards,

Yves.

The other week I visited Tirana, Albania. In many ways it is not unlike other parts of Europe, which surprised me because its “opening up” to the rest of the world is of such such recent history (under Hoxha it was the North Korea of Europe).
One thing that certainly has changed since Hoxha’s time is city traffic. In those days there were less than 2000 cars in the whole of Albania, and look at it now:

This is a video I took from the Sky Tower on a monday at three o’clock.
Depicted are the intersections of the Bulevardi Gjergj Fishta and the Bulevardi Bajram Curri with the Rruga Dëshmorët e 4 Shkurtit. Top-right: the southwest corner of the Parku Rinia, and right across the middle of the picture the Lanë stream which is more of an open sewer, actually.

Video on this page is embedded through the HTML5 Video element, with two video streams.
I did the VP8 WebM with ffmpeg:

ffmpeg -i raw.avi -f webm -vcodec libvpx -acodec libvorbis -aq 3 -b 800 -y TiranaTraffic.webm

The specified bitrate seems to be ignored by the encoder. Libvpx support in ffmpeg is rather new so I guess this will be resolved in the near future.
The fallback Ogg Theora I did with ffmpeg2theora:

ffmpeg2theora -o TiranaTraffic.ogv --two-pass --optimize -V 1200 raw.avi