Writing to Then Reading from a Memorystream

Writing to then reading from a MemoryStream

Three things:

Don't close the StreamWriter. That will close the MemoryStream. You do need to flush the writer though.
Reset the position of the stream before reading.
If you're going to write directly to the stream, you need to flush the writer first.

So:

using (var stream = new MemoryStream())
{
    var sw = new StreamWriter(stream);
    sw.Write("{");

    foreach (var kvp in keysAndValues)
    {
        sw.Write("'{0}':", kvp.Key);
        sw.Flush();
        ser.WriteObject(stream, kvp.Value);
    }    
    sw.Write("}");            
    sw.Flush();
    stream.Position = 0;

    using (var streamReader = new StreamReader(stream))
    {
        return streamReader.ReadToEnd();
    }
}

There's another simpler alternative though. All you're doing with the stream when reading is converting it into a string. You can do that more simply:

return Encoding.UTF8.GetString(stream.GetBuffer(), 0, (int) stream.Length);

Unfortunately MemoryStream.Length will throw if the stream has been closed, so you'd probably want to call the StreamWriter constructor that doesn't close the underlying stream, or just don't close the StreamWriter.

I'm concerned by you writing directly to the the stream - what is ser? Is it an XML serializer, or a binary one? If it's binary, your model is somewhat flawed - you shouldn't mix binary and text data without being very careful about it. If it's XML, you may find that you end up with byte-order marks in the middle of your string, which could be problematic.

Writing String to Stream and reading it back does not work

After you write to the MemoryStream and before you read it back, you need to Seek back to the beginning of the MemoryStream so you're not reading from the end.

UPDATE

After seeing your update, I think there's a more reliable way to build the stream:

UnicodeEncoding uniEncoding = new UnicodeEncoding();
String message = "Message";

// You might not want to use the outer using statement that I have
// I wasn't sure how long you would need the MemoryStream object    
using(MemoryStream ms = new MemoryStream())
{
    var sw = new StreamWriter(ms, uniEncoding);
    try
    {
        sw.Write(message);
        sw.Flush();//otherwise you are risking empty stream
        ms.Seek(0, SeekOrigin.Begin);

        // Test and work with the stream here. 
        // If you need to start back at the beginning, be sure to Seek again.
    }
    finally
    {
        sw.Dispose();
    }
}

As you can see, this code uses a StreamWriter to write the entire string (with proper encoding) out to the MemoryStream. This takes the hassle out of ensuring the entire byte array for the string is written.

Update: I stepped into issue with empty stream several time. It's enough to call Flush right after you've finished writing.

MemoryStream have one thread write to it and another read

You can't use a Stream with seeking capabilities from 2 threads simultaneous since a Stream is state full. e.g. A NetworkStream has 2 channels, one for reading and one for writing and therefore can't support seeking.

If you need seeking capabilities, you need to create 2 streams, one for reading and one for writing respectively. Else you can simply create a new Stream type which allows reading and writing from a underlying memory stream by taking exclusive access to the underlying stream and restore its write/read position. A primitive example of that would be:

class ProducerConsumerStream : Stream
{
    private readonly MemoryStream innerStream;
    private long readPosition;
    private long writePosition;

    public ProducerConsumerStream()
    {
        innerStream = new MemoryStream();
    }

    public override bool CanRead { get { return true;  } }

    public override bool CanSeek { get { return false; } }

    public override bool CanWrite { get { return true; } }

    public override void Flush()
    {
        lock (innerStream)
        {
            innerStream.Flush();
        }
    }

    public override long Length
    {
        get 
        {
            lock (innerStream)
            {
                return innerStream.Length;
            }
        }
    }

    public override long Position
    {
        get { throw new NotSupportedException(); }
        set { throw new NotSupportedException(); }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        lock (innerStream)
        {
            innerStream.Position = readPosition;
            int red = innerStream.Read(buffer, offset, count);
            readPosition = innerStream.Position;

            return red;
        }
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        throw new NotSupportedException();
    }

    public override void SetLength(long value)
    {
        throw new NotImplementedException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        lock (innerStream)
        {
            innerStream.Position = writePosition;
            innerStream.Write(buffer, offset, count);
            writePosition = innerStream.Position;
        }
    }
}

Write and read from a stream asynchronously

Is there another option for this?

The code is currently downloading to a memory stream and then writing it out to disk. If that's all you need to do with it, then passing a file stream instead of a memory stream should be sufficient.

What I am struggling with is the reading since it seems streams (MemoryStream in this case) does not support simultaneous reading and writing.

This is correct. No built-in stream types support simultaneous reads and writes. One reason is that there's a notion of a single "position" that is updated by both reads and writes.

It should be possible to write a concurrent stream type. You'd need to handle concurrent access as well as having two positions rather than one, and some operations might not be supported for concurrent streams. I've thought about writing a type like this a few times, but didn't feel it was sufficiently useful in a world where Pipelines, Channels, Dataflow, and async streams already exist.

So, I'd say:

If possible, just pass it a file stream and get rid of the memory stream completely.
Otherwise, explore the GCP API and see if there's a non-stream-based solution you can use to download.
If the GCP API must download to a stream and your code must do in-memory processing during the save, then you'll need to write a concurrent stream type.

Serializing to stream and then reading from stream

The issue that the example code is exhibiting is double disposing (the StreamReader and the XmlWriter both dispose their BaseStream[MemoryStream]).

Putting the StreamReader into the top block of usings fixes the issue.

But in the end I didn't need to StreamReader to get the text out and went with the following:

        public static string SerializeToString(T obj)
        {
            XmlSerializer serializer = new XmlSerializer(typeof(T));

            using (var memoryStream = new MemoryStream())
            using (var writer = XmlWriter.Create(memoryStream))
            {
                serializer.Serialize(writer, obj);
                return Encoding.UTF8.GetString(memoryStream.GetBuffer(), 0, (int)memoryStream.Length);
            }
        }

StreamReader from MemoryStream

You can initialize the MemoryStream from an array to begin with; then you don't have to write anything to a stream. Since Base64, by definition, is pure ascii text, you just convert the input string to bytes using ASCII encoding.

Though, if you're parsing CSV, there are better output options than just text read line by line. Technically, CSV format can contain line breaks inside fields, a feature supported by pretty much anything that writes CSV from spreadsheet files (like MS Excel). To support this, the line-by-line-reading approach is too simple. The .Net framework contains a native CSV reader, though, albeit hidden quite well in the Microsoft.VisualBasic classes. Since .Net is one framework, though, there's nothing preventing you from adding the reference and using it in C#. The class is TextFieldParser, from Microsoft.VisualBasic.FileIO.

public static List<String[]> ParseBase64Csv(String data, Encoding encoding, Char separator, Boolean ignoreEmptyLines)
{
    List<String[]> splitLines = new List<String[]>();
    using (MemoryStream ms = new MemoryStream(Encoding.ASCII.GetBytes(data)))
    using (FromBase64Transform tr = new FromBase64Transform(FromBase64TransformMode.IgnoreWhiteSpaces))
    using (CryptoStream cs = new CryptoStream(ms, tr, CryptoStreamMode.Read))
    using (StreamReader sr = new StreamReader(cs, encoding))
    using (TextFieldParser tfp = new TextFieldParser(sr))
    {
        tfp.TextFieldType = FieldType.Delimited;
        tfp.Delimiters = new String[] { separator.ToString() };
        while (true)
        {
            try
            {
                String[] curLine = tfp.ReadFields();
                if (curLine == null)
                    break;
                if (ignoreEmptyLines && (curLine.Length == 0 || curLine.All(x => String.IsNullOrEmpty(x) || String.IsNullOrEmpty(x.Trim()))))
                    continue;
                splitLines.Add(curLine);
            }
            catch (MalformedLineException mfle)
            {
                // do something with errors here.
            }
        }
    }
    return splitLines;
}

MemoryStream is not writing anything to the file

After writing to MemoryStream it's Position is at the end. You need to set position back to the beginning:

WriteToStream(ms);
ms.Seek(0, SeekOrigin.Begin); // < here
CopyFromStream(ms, "database.dat");

Also, Stream class already has CopyTo method so there is no need to reinvent it (except maybe you are on some very old framework vesrion where it's not available?):

WriteToStream(ms);
ms.Seek(0, SeekOrigin.Begin); // < here
using (var fs = File.Open("database.dat", FileMode.OpenOrCreate)) {
    ms.CopyTo(fs);
}

Writing a memory stream to a file

There is a very handy method, Stream.CopyTo(Stream).

using (MemoryStream ms = new MemoryStream())
{
    StreamWriter writer = new StreamWriter(ms);

    writer.WriteLine("asdasdasasdfasdasd");
    writer.Flush();

    //You have to rewind the MemoryStream before copying
    ms.Seek(0, SeekOrigin.Begin);

    using (FileStream fs = new FileStream("output.txt", FileMode.OpenOrCreate))
    {
        ms.CopyTo(fs);
        fs.Flush();
    }
}

Also, you don't have to close fs since it's in a using statement and will be disposed at the end.

Writing to Then Reading from a Memorystream