the question

from Yves <theYinYeti@yeti.selfip.net>
to wicher@gavagai.eu
date Wed, Sep 15, 2010 at 16:06
subject Socat

Dear mister Minnaard,

I hope you don’t mind me asking you for help. By chance, I saw your blog post:
http://smorgasbord.gavagai.nl/2010/01/socat-access-your-x-servers-domain-socket-over-tcp/

It happens I have a problem with socat on my Sheevaplug server, and it seems you have good knowledge of this software.

Following http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, I use socat to tunnel DNS through SSH, so as to have transparent DNS resolution, to be used with a transparent proxy (NAT rules + tinyproxy, backed by cntlmd, backed by corporate proxy).

My problem is that the number of socat PIDs on both the client and the server keeps growing. I’ve had floods of “accept: too many open files” messages, and I suspect socat is the culprit (at this point, a ps -ef shows almost a thousand of socat PIDs!); especially since the transparent DNS stops working after this flood of messages.

More specifically, on the client, I run:

• dnsmasq with “server=127.0.0.1#1053” as a back-end

• socat -ly udp4-listen:1053,reuseaddr,fork tcp:127.0.0.1:1054

• ssh -L 1053:208.67.222.222:53 -L 1054:127.0.0.1:1054 \
  -Tfx remoteuser@remoteserver \
  'socat -lf /dev/null tcp4-listen:1054,reuseaddr,fork \
  UDP:208.67.222.222:53'

  (208.67.222.222 is OpenDNS)

Besides, on localhost (local client), ssh is configured this way:

Host remoteserver
Port 443
ServerAliveInterval 60
ProxyCommand /usr/bin/proxytunnel -v -p corporateproxy:8080 -P login:passwd -d %h:%p

Do you have an idea on how I could make socat behave better?
I thank you beforehand for any advice you could provide.
Sincerely,

Yves.

the reply

from Wicher Minnaard <wicher@gavagai.eu>
to Yves <theYinYeti@yeti.selfip.net>
date Thu, Sep 16, 2010 at 02:19
subject Re: Socat

Dear mister Yves,

On Wed, Sep 15, 2010 at 16:06, Yves <yves@yeti.selfip.net> wrote:

Dear mister Minnaard,

I hope you don’t mind me asking you for help. By chance, I saw your blog post:
http://smorgasbord.gavagai.nl/2010/01/socat-access-your-x-servers-domain-socket-over-tcp/

It happens I have a problem with socat on my Sheevaplug server, and it seems
you have good knowledge of this software.

Thank you for your compliment. Actually, I do not know socat all too well, but I know a thing or two about networking.

Following http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, I use socat to tunnel DNS through SSH, so as to have transparent DNS resolution, to be used with a transparent proxy (NAT rules + tinyproxy, backed by cntlmd, backed by
corporate proxy).

Sounds like just another day at the office ;-)
Interesting setup.

My problem is that the number of socat PIDs on both the client and the server keeps growing. I’ve had floods of “accept: too many open files” messages, and I suspect socat is the culprit (at this point, a ps -ef shows almost a thousand of socat PIDs!); especially since the transparent DNS stops working after this flood of messages.

• dnsmasq with “server=127.0.0.1#1053” as a back-end

• socat -ly udp4-listen:1053,reuseaddr,fork tcp:127.0.0.1:1054

• ssh -L 1053:208.67.222.222:53 -L 1054:127.0.0.1:1054 \
  -Tfx remoteuser@remoteserver \
  'socat -lf /dev/null tcp4-listen:1054,reuseaddr,fork \
  UDP:208.67.222.222:53'

  (208.67.222.222 is OpenDNS)

Sidenote: Google provides a solid DNS service nowadays. I don’t know what they do with the data, though the same could be said about OpenDNS.
See http://code.google.com/speed/public-dns/


Host remoteserver
Port 443
ServerAliveInterval 60
ProxyCommand /usr/bin/proxytunnel -v -p corporateproxy:8080 -P login:passwd -d %h:%p


So it’s DNS over UDP over TCP over SSH over proxied HTTPS, or something. I’m losing track of all the layers here.

Do you have an idea on how I could make socat behave better?
I thank you beforehand for any advice you could provide.

Well you got me curious now.
I managed to reproduce your problem in a simplified setup:

- on host w00tage, setup the nameserver façade:
socat udp4-listen:1053,reuseaddr tcp:localhost:1054

- from host w00tage, tunnel the TCP connection over SSH to the host hosting the to-be-proxied nameserver:
ssh -L 1054:localhost:1054 priv

- now, on this host priv, connect the TCP socket to the UDP socket where the nameserver is listening
socat tcp4-listen:1054,fork,reuseaddr UDP:localhost:53

- on w00tage, do queries on localhost:1054:
dig @localhost -p1053 google.fr

Dig is part of the bind-tools package and greatly helps in diagnosing DNS problems.
The first dig will probably return. Subsequent digs may take seconds to return. And they leave stray socat processes behind on both w00tage and priv. Kill the strays on w00tage, and the ones on priv dissolve. So the local ones (on w00tage) tie up the remote ones. The latter are not the problem, just a symptom. You can keep on querying, doing more digs, until your file descriptors run out.

A stab at what is going on here: UDP is stateless, and socat is ignorant of protocol data (as it should be). A DNS query takes one UDP packet. But socat does not know we’re doing DNS. And since UDP is stateless, there’s no ‘bye’ in the conversation. So socat doesn’t know when to stop waiting for more. Dig does, though – it recognizes a DNS answer when it sees one. So dig returns (but why the delays? socat buffers and timeouts maybe?) but socat doesn’t. It’s still waiting for more.
Now since we’re forking socat for every new independent socketpair opened (which is what happens when you do a query), we get a lot of those hanging socats.

I had a look at the URL you sent me, http://zarb.org/~gc/html/udp-in-ssh-tunneling.html, to see how they are getting it to work and it appears there is a ‘UDP-RECVFROM’ address type in socat. From its manpage:
————
UDP-RECVFROM: Creates a UDP socket on [UDP service] using UDP/IP version 4
or 6 depending on option pf. It receives one packet from an
unspecified peer and may send one or more answer packets to that peer.
This mode is particularly useful with fork option where each arriving
packet – from arbitrary peers – is handled by its own sub process.
This allows a behaviour similar to typical UDP based servers like
ntpd or named. This address works well with socat UDP-SENDTO address
peers.
————

Hey, explicit mentioning of nameservers there. Receive one packet, don’t wait for more, send it off, collect answer, send that back.
On host w00tage, I run the façade thusly:

socat udp4-recvfrom:1053,fork,reuseaddr tcp:localhost:1054

And that appears to work indeed. But wait, how does socat (on priv) know when to stop trying to collect answer packets?
I think it’s socat timeouts, but I’m not 100% sure. What leads me to think that it’s timeouts is that when I stress-test this setup from w00tage:

while true; do dig @localhost -p1053 sheeva; done
(for ’sheeva’ the nameserver never has to do recursion)
I have about 30 forked-off query-listeners when I do a pgrep socat | wc -l on w00tage. But I have about 400 on priv. That number goes up and down and the pids change, so they do not *accumulate* but rather seem to hang about until they decide their job is done. But to be sure I’d have to study the source. Or ask the socat mailing list.

Anyway, problem solved!

I was happy to help you and I got some exercise out of it. Any reason why you didn’t post this question as a comment on my blog? Maybe it would be a bit off-topic, I can see that.
Would you mind if I published this?

Regards and good luck, Wicher Minnaard.

it works

from Yves <theYinYeti@yeti.selfip.net>
to wicher@gavagai.eu
date Thu, Sep 16, 2010 at 10:01
subject Re: Socat

Dear mister Minnaard,

All in all, I was just careless in adapting the “alternative solution with socat” to my setup.

I thank you for your analysis and tests; I learnt a few things following them. And I certainly wouldn’t mind at all if you were to publish this.
However, if you do so, please change the e-mail to “theYinYeti@yeti.selfip.net” (the Sheevaplug :-) ), and my name to simply “Yves”; thank you. As for the web site (I see there’s a field for it), it is “http://yeti.selfip.net/gablin.php” (a bit lame, but I may find time someday to update it…)

As you said, I had the feeling this was a bit off-topic. But now, I see how this can shed some light on advanced socat usage.
–
Regards,

Yves.

This happens to me regularly. Someone brings a machine along and I want to display some app, running on my machine, on their display. Networked X11 to the rescue, you say? No, their X11 server is started with ‘-nolisten TCP’ wich is the default on most modern Linux distros. Sadly, the TCP socket can’t be enabled ‘in-flight’ — if you decide you do fancy a TCP socket after all, you’ll have to restart your X server which may be a pain if you’re in the middle of something (besides, restarting is just plain uncool).
But there is a way to expose the Unix domain socket as a TCP socket, with the help of socat. The following examples all use bash, so if you run a different shell (if you don’t know, you probably aren’t) you may need to define environment variables differently.

Braindead Proof of Concept (BPOC)

Situation: You want to display an application running on a machine called w00t on another machine, called bling. There’s an X11 server running on bling, but it’s not configured to listen on any TCP socket. DNS is properly setup, so if you ping w00t from bling, you get replies from bling’s IP, and vice versa.

1. On bling, find the domain socket of bling’s X11 server. Have a look in /tmp/.X11-unix/. The socket’s name usually reflects its X server display number (which you can determine by running echo $DISPLAY in an xterm).
2. On bling, run something along the lines of
   socat TCP-LISTEN:6066 UNIX-CONNECT:/tmp/.X11-unix/X0
   This will open up TCP port 6066 on all of bling’s network interfaces, connecting it to the Unix domain socket of the X server.
3. In an xterm on bling, run xhost +. You have now opened up your X11 server to the whole wide world, a silly thing to do. Anyone with access to the TCP socket can now read your keystrokes, read your window contents, click your mouse buttons…
4. In an xterm on w00t, run DISPLAY="bling:66" xclock. You may have noticed that 66 = 6066 – 6000 and indeed, by convention the TCP port number for a certain display is its display number + 6000. Anyhow…. yay, a clock! It’s displayed on bling, but running on w00t.

Improvements

• You may have noticed that in the BPOC, you can use the display on bling only once. socat will allow only one client, and will exit once that client exits. In some situations, you may consider that a feature (it’s a one-time access grant), but in others you may not. If you want a reusable TCP socket, run something along the lines of
  socat TCP-listen:6066,fork,reuseaddr UNIX-CONNECT:/tmp/.X11-unix/X0 which forks off a socat process for every TCP connection.
• You may not want to expose a TCP socket on all interfaces. Maybe you only want to expose a socket on the LAN interface, or on the localhost interface (and wrap the packets in an SSH tunnel). Well, you can, using the ‘bind’ option:
  socat TCP-LISTEN:6066,bind=localhost UNIX-CONNECT:/tmp/.X11-unix/X0
  Now tunnel it over SSH. On w00t, run ssh -L 6011:localhost:6023 bling. Now localhost:6011 on woot is actually localhost:6023 on bling which is actually /tmp/.X11-unix/X0 on bling. So on w00t you can start an xclock with its display on bling by running DISPLAY="localhost:11" xclock.
• xhost + from the BPOC is braindead indeed. There are a couple things you could have done instead, there are good ways of tightening up your authorization scheme.
  • First off, you don’t really need to run xhost + if you properly set up X11 cookies, which you should. Here are some examples on using the xauth scheme, but take note: xauth generate will probably not work on recent X11 releases since the XSECURITY extension is disabled by default. Just use the same cookies on the client and the server.
  • Run xhost +w00t. That’s host-based authentication, which is stupid, but not as stupid as no authorization at all. Any user on w00t can now connect.
  • Suppose that on bling (of course!) you’d run xhost +SI:localuser:theuser with ‘theuser’ being the userID of the unix-user running the socat instance. Now from the point of view of the X server, any client connecting through socat will be coming from ‘theuser’ and will therefore be allowed access. Entertaining, but not much different from just running xhost +. It is something to keep in mind though! Many distros by default add the unix-user that started the X server to the authorization list. That user does not need a cookie. If you run socat as that user you will have the effect of running xhost + even if you run xhost -.
  • Just run a nested X11 server, such as Xnest or Xephyr. This way you put untrusted users in a sandbox, preventing them from snooping your keyboard and windows. It’s the X11 equivalent of a chroot.