Securing Udp - Openssl or Gnutls or ...

Securing UDP - OpenSSL or GnuTls or ...?

I've found the following facts about the libraries and DTLS.

There is another lib with DTLS support - CyaSSL, but it supports DTLS only in test mode for now.
Although RFC 4347 dates from Apr, 2006, the OpenSSL supports DTLS since 2005 (v0.9.8). Many Linux distribs include this version. OpenSSL API looks ugly a little, but it seems like DTLS implementation is stable.
GnuTls supports DTLS since 2011 (v3.0.0). Looks like no Linux includes this version yet. (For example, Ubuntu 11.04 uses v2.8.6, Ubuntu 11.10 is going to use v2.10.5, not v3.0.0.) There is no information about when v3.0 will be used. It can be built manually, however it depends on too many additional libraries which may have no native support in some distribs.
It looks like all of these libraries can be used on other platforms (e.g. Windows).
Known OpenSSL issue: OpenSSL has compression enabled by default for DTLS, but it shouldn't be. OpenSSL v0.9.8 API doesn't provide any method to disable compression. The method should be implemented manually.

SUMMARY:

Speaking about usability, personally I would prefer GnuTls API, but at the time OpenSSL looks more preferable to use.

How Should I Implement Security On UDP Socket

You are looking for DTLS, the Datagram TLS.

It is like the TLS protocol that you know from HTTPS and various other secure point-to-point communication links, but it is implemented over UDP. You will find it already implemented in various libraries including GnuTLS and OpenSSL.

From the security point-of-view, one major difference between TLS and DTLS is that TLS defines an ill-formed message as an unrecoverable error, whereas DTLS specifically allows the connection to continue in this case. This makes the protocol more sensitive to even slight coding errors (think Lucky Thirteen), so you had better not try to implement it yourself.

Secure UDP Socket Programming

Is there a way I can prevent packets from arriving from a certain source, or any source besides the correct one?

Yes. Just connect() the socket to that correct source. Then UDP will filter out all datagrams from other addresses. See man 2 connect, the paragraph about SOCK_DGRAM sockets.

Adding SSL support to existing TCP & UDP code?

SSL is very complex, so you're going to want to use a library.

There are several options, such as Keyczar, Botan, cryptlib, etc. Each and every one of those libraries (or the libraries suggested by others, such as Boost.Asio or OpenSSL) will have sample code for this.

Answering your second question (how to integrate a library into existing code without causing too much pain): it's going to depend on your current code. If you already have simple functions that call the Winsock or socket methods to send/receive ints, strings, etc. then you just need to rewrite the guts of those functions. And, of course, change the code that sets up the socket to begin with.

On the other hand, if you're calling the Winsock/socket functions directly then you'll probably want to write functions that have similar semantics but send the data encrypted, and replace your Winsock calls with those functions.

However, you may want to consider switching to something like Google Protocol Buffers or Apache Thrift (a.k.a. Facebook Thrift). Google's Protocol Buffers documentation says, "Prior to protocol buffers, there was a format for requests and responses that used hand marshalling/unmarshalling of requests and responses, and that supported a number of versions of the protocol. This resulted in some very ugly code. ..."

You're currently in the hand marshalling/unmarshalling phase. It can work, and in fact a project I work on does use this method. But it is a lot nicer to leave that to a library; especially a library that has already given some thought to updating the software in the future.

If you go this route you'll set up your network connections with an SSL library, and then you'll push your Thrift/Protocol Buffer data over those connections. That's it. It does involve extensive refactoring, but you'll end up with less code to maintain. When we introduced Protocol Buffers into the codebase of that project I mentioned, we were able to get rid of about 300 lines of marshalling/demarshalling code.

Query-Regarding-Gnutls-Build

You can make use of callgrind to profile and see if any api in gnutls is called repeatedly. Though this looks an old one, but you can find how callgrind was used to find the issue.

what is a good way to secure Cap'n Proto RPC network traffic?

Yes, Cap'n Proto's two-party protocol (the only one provided currently) should work great with stunnel, since it's a simple TCP-based transport. You will need to run both a stunnel client and a server, of course, but otherwise this should be straightforward to set up. You could also use SSH port forwarding or a VPN to achieve a similar result.

(Note that stunnel itself has nothing to do with HTTPS per se, but is often used to implement HTTPS because HTTP is also a simple TCP protocol and HTTPS is the same protocol except on TLS. In the Cap'n Proto case, Cap'n Proto replaces HTTP. So you're creating Cap'nProto-S, I guess.)

Another option is to implement the kj::AsyncIoStream abstract interface directly based on a TLS library like OpenSSL, GnuTLS, etc. Cap'n Proto's RPC layer will allow you to provide an arbitrary implementation of kj::AsyncIoStream as its transport (via interfaces in capnp/rpc-twoparty.h). Unfortunately, many TLS libraries have pretty ugly interfaces and so this may be hard to get right. But if you do write something, please contribute it back to the project as this is something I'd like to have in the base library.

Eventually we plan to add an official crypto transport to Cap'n Proto designed to directly support multi-party introductions (something Cap'n Proto actually doesn't do yet, but which I expect will be a killer feature when it's ready). I expect this support will appear some time in 2016, but can't make any promises.

Couldn't open https stream - protocol not found (ffmpeg with openssl)

When I removed next line

--disable-protocols \

It began work.
This is weird thing coz in my script there is one more line that will enable all required for my purposes protocols:

--enable-protocol=file,ftp,http,https,httpproxy,hls,mmsh,mmst,pipe,rtmp,rtmps,rtmpt,rtmpts,rtp,sctp,srtp,tcp,udp \

Also, I've updated openssl lib version to 1.0.2j.
Anyway, for me this is the solution.
Thanks @Mulvya for help.

Securing Udp - Openssl or Gnutls or ...