Manipulating NTFS alternate data streams in C# with the CodeFluent Runtime Client

Have you already heard about NTFS alternate streams? Also known as named streams or ADS (Alternate Data Streams).
Well it is a useful feature in NTFS storage systems. It expands the concept of file and data streams.

Alternate Data Streams

When working with NTFS files, the main data stream (or unnamed stream) is the central element of the file. When you create a file, a main stream is created. When you create an alternate stream and the main stream does not exist it is created, if you delete the main stream the whole file is deleted (so the existing alternate streams).
When you read a file or you write in to a file you are working with the main stream by default.

Alternate streams follows the syntax: “filename.ext:alternateName”

You can store any kind of data in an ADS (as you can do it with the unnamed stream), so you can store binary data, text data, an image, a video and even an executable file.

Let’s make quick test.

Open a command line console (cmd.exe).
Create a text file and write some content in it:

Writing in to the main data stream

Let’s read the content:

Reading the main stream

Nothing extraordinary, we get the main data stream from our file.
Now let’s try to write in to an alternate data stream (this will create the alternate stream if it does not exist).

Writing in to an Alternate Data Stream

You have created an alternate data stream called “hide” right on our file “test.txt”, this will not have any incidence with your main data stream.
To ensure that our alternate data stream “hide” has been correctly created we will try to read it.

Reading an Alternate Data Stream content

And to prove that our main data stream is still there, let’s read the main stream.

Reading the main stream

We have made some tests only with “text” streams but a stream can be also an image, an executable file and all other kind of stream a file container can host.
What about manipulating Alternate Data Streams with C#? 
Well, this feature is unfortunately not available in .NET, we would need to call native methods if we want to manipulate alternate data streams.
So we can build some nice native method wrappers in order to manipulate alternate data streams or we can use the CodeFluent Entities Runtime Client.
CodeFluent Runtime Client is a free library that provides very useful and powerful helpers like:

• XML utilities
• IO utilities
• Type conversion helpers
• JSON utilities
• … and many other

You can easily install the CodeFluent Runtime Client from Nugget.

PM> Install-package CodeFluentRuntimeClient

Using the CodeFluent Entities Client Runtime to manipulate alternate data streams is as easy as manipulate all well-known file streams.
We will use the NtfsAlternateStream class which is located in theCodeFluent.Runtime.BinaryServices namespace, it provides some static helper methods to manipulate alternate streams as they were “regular” streams (open, create, read, write, enumerate, delete…).
Let’s take a look to some useful methods to manipulate alternate data streams (ADS):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

//Create a stream supporting ADS syntax FileStream stream = NtfsAlternateStream.Open("test.txt:hide", FileAccess.Write, FileMode.OpenOrCreate, FileShare.None); stream.Close();   //Writing in to an ADS NtfsAlternateStream.WriteAllText("test.txt:hide", "Secret content");   //Reading data from an ADS string text = NtfsAlternateStream.ReadAllText("test.txt:hide");   //Enumerating all the ADS in test.txt IEnumerable adsStreams = NtfsAlternateStream.EnumerateStreams("test.txt"); foreach (NtfsAlternateStream ads in adsStreams) {     Console.WriteLine(ads.Name); }   //This will not delete the test.txt file NtfsAlternateStream.Delete("test.txt:hide");

A concrete example of how ADS are used in Windows is when you download a file from the Internet. When I open a file downloaded from the Internet with Word (2013) I receive a warning telling me that the file might not be secure.

Word Protected View

How does Word know that I downloaded the file from the Internet? Well, every time you download a file, Windows set an ADS called “:Zone.Identifier” containing some data related to the origin of the file. Let’s confirm that.

1 2 3

string altFileName = @"C:\Users\pablo\Downloads\someDoc.docx:Zone.Identifier"; string content = NtfsAlternateStream.ReadAllText(altFileName); Console.WriteLine(content);

What we get is:

ZoneTransfer ZoneId

This value “ZoneId=3” means that the file has “Internet” as origin.

If we delete the “:Zone.Identifier” ADS from the file:

1 2 3

string altFileName = @"C:\Users\pablo\Downloads\someDoc.docx:Zone.Identifier"; //this will not delete the file NtfsAlternateStream.Delete(altFileName);

Now we don’t receive a warning when trying to open the file, Word has no information from the file origin.
Alternate data streams are nice but they have some limitations. As we have said, ADS are only supported in NTFS file storage systems so what happen if you copy a file containing ADSs to another file system (FAT file system, USB drive, CD/DVD, network transfer…)? Well, you will lose all your ADS!

Avoid writing important or critical data to alternate data streams. ADS are not supported in not NTFS file systems.

Some ideas where ADS might be useful:

• If you are writing a program to edit images it will be nice to keep the original image (or even all the modification history) so the user can undo some changes. Instead of keeping separate files you can write all the image versions in the same file using ADS, e.g.image.jpg:original, image.jpg:v1…
• You can store thumbnails for graphical files.
• Imagine you wrote a “reader” application, you can keep some information like: font size, current page, background color… in the file itself.

I am sure you can image other practical and fun uses for Alternate Data Streams, it would be great if you share it with us .

Regards.
Pablo Fernandez Duran

線程鎖的概念函數EnterCriticalSection和LeaveCriticalSection的用法

使用結構CRITICAL_SECTION 需加入頭文件#include 「afxmt.h」

定義一個全局的鎖 CRITICAL_SECTION的實例
和一個靜態全局變量

CRITICAL_SECTION cs;//可以理解為鎖定一個資源
static int n_AddValue = 0;//定義一個靜態的全部變量n_AddValue

創建兩個線程函數，代碼實現如下：

代碼

//第一個線程
UINT FirstThread(LPVOID lParam)
{
     EnterCriticalSection(&cs);//加鎖 接下來的代碼處理過程中不允許其他線程進行操作，除非遇到LeaveCriticalSection
     for(int i = 0; i<10; i++){       
         n_AddValue ++;
         cout << "n_AddValue in FirstThread is "<<n_AddValue <<endl;       
     }
     LeaveCriticalSection(&cs);//解鎖 到EnterCriticalSection之間代碼資源已經釋放了，其他線程可以進行操作   
     return 0;
}//第二個線程
UINT SecondThread(LPVOID lParam)
{
    EnterCriticalSection(&cs);//加鎖
    for(int i = 0; i<10; i++){       
        n_AddValue ++;       
        cout << "n_AddValue in SecondThread is "<<n_AddValue <<endl;   
       
    }
    LeaveCriticalSection(&cs);//解鎖
    return 0;
}

在主函數添加以下代碼

代碼

 int _tmain(int argc, TCHAR* argv[], TCHAR* envp[])
 {
    int nRetCode = 0;
 
    // 初始化 MFC 並在失敗時顯示錯誤
     if (!AfxWinInit(::GetModuleHandle(NULL), NULL, ::GetCommandLine(), 0))
    {
         // TODO: 更改錯誤代碼以符合您的需要
         _tprintf(_T("錯誤: MFC 初始化失敗\n"));
        nRetCode = 1;
     }
     else
    {
        InitializeCriticalSection(&cs);//初始化結構CRITICAL_SECTION
 
        CWinThread *pFirstThread,*pSecondThread;//存儲函數AfxBeginThread返回的CWinThread指針
       
        pFirstThread  = AfxBeginThread(FirstThread,LPVOID(NULL));//啟動第一個線程
        pSecondThread = AfxBeginThread(SecondThread,LPVOID(NULL));//啟動第二個線程
  
        HANDLE hThreadHandle[2];//        hThreadHandle[0] = pFirstThread->m_hThread;
        hThreadHandle[1] = pSecondThread->m_hThread;
 
        //等待線程返回
        WaitForMultipleObjects(2,hThreadHandle,TRUE,INFINITE);       
    }
 
    return nRetCode;
}

輸出：

n_AddValue in FirstThread is 1
n_AddValue in FirstThread is 2
n_AddValue in FirstThread is 3
n_AddValue in FirstThread is 4
n_AddValue in FirstThread is 5
n_AddValue in FirstThread is 6
n_AddValue in FirstThread is 7
n_AddValue in FirstThread is 8
n_AddValue in FirstThread is 9
n_AddValue in FirstThread is 10
n_AddValue in SecondThread is 11
n_AddValue in SecondThread is 12
n_AddValue in SecondThread is 13
n_AddValue in SecondThread is 14
n_AddValue in SecondThread is 15
n_AddValue in SecondThread is 16
n_AddValue in SecondThread is 17
n_AddValue in SecondThread is 18
n_AddValue in SecondThread is 19
n_AddValue in SecondThread is 20

如果把兩個線程函數中的EnterCriticalSection和LeaveCriticalSection位置移到for循環中去，線程的執行順序將會改變
輸出也就跟著改變，如：

代碼

//第一個線程
 UINT FirstThread(LPVOID lParam)
{
    
    for(int i = 0; i<10; i++){
        EnterCriticalSection(&cs);//加鎖 鎖移到for循環內部裡
        n_AddValue ++;
        cout << "n_AddValue in FirstThread is "<<n_AddValue <<endl;   
        LeaveCriticalSection(&cs);//解鎖 
    }   
    return 0;
}
  
 //第二個線程
UINT SecondThread(LPVOID lParam)
{
   
    for(int i = 0; i<10; i++){   
        EnterCriticalSection(&cs);//加鎖
        n_AddValue ++;       
        cout << "n_AddValue in SecondThread is "<<n_AddValue <<endl;
        LeaveCriticalSection(&cs);//解鎖       
     }
     return 0;
 }

其他代碼不變，輸出的結果如下：

n_AddValue in FirstThread is 1
n_AddValue in SecondThread is 2
n_AddValue in FirstThread is 3
n_AddValue in SecondThread is 4
n_AddValue in FirstThread is 5
n_AddValue in SecondThread is 6
n_AddValue in FirstThread is 7
n_AddValue in SecondThread is 8
n_AddValue in FirstThread is 9
n_AddValue in SecondThread is 10
n_AddValue in FirstThread is 11
n_AddValue in SecondThread is 12
n_AddValue in FirstThread is 13
n_AddValue in SecondThread is 14
n_AddValue in FirstThread is 15
n_AddValue in SecondThread is 16
n_AddValue in FirstThread is 17
n_AddValue in SecondThread is 18
n_AddValue in FirstThread is 19
n_AddValue in SecondThread is 20

個人認為在函數EnterCriticalSection和LeaveCriticalSection中間的代碼執行過程不會被其他線程干攏或者這麼講不允許其他線程中
的代碼執行。這樣可以有效防止一個全局變量在兩個線程中同時被操作的可能性

Thread Pooling in C#

Thread Pooling

Thread pooling is the process of creating a collection of threads during the initialization of a multithreaded application, and then reusing those threads for new tasks as and when required, instead of creating new threads. Then every process has some fixed number of threads depending on the amount of memory available, those threads are the need of the application but we have freedom to increase the number of threads. Every thread in the pool has a specific given task. The thread returns to the pool and waits for the next assignment when the given task is completed.

Usually, the thread pool is required when we have number of threads are created to perform a number of tasks, in this organized in a queue. Typically, we have more tasks than threads. As soon as a thread completes its task, it will request the next task from the queue until all tasks have been completed. The thread can then terminate, or sleep until there are new tasks available.

Creating thread pooling

The .Net framework library included the "System.Threading.ThreadPool" class. it was so easy to use.You need not create the pool of threads, nor do you have to specify how many consuming threads you require in the pool. The ThreadPool class handles the creation of new threads and the distribution of the wares to consume amongst those threads.

There are a number of ways to create the thread pool:

• Via the Task Parallel Library (from Framework 4.0).
• By calling ThreadPool.QueueUserWorkItem.
• Via asynchronous delegates.
• Via BackgroundWorker.

Entering the Thread Pool via TPL

The task parallel library provide the task class for enter the thread pool easy. The task class is the part of .Net Framework 4.0 .if you're familiar with the older constructs, consider the nongeneric Task class a replacement for ThreadPool.QueueUserWorkItem, and the generic Task<TResult> a replacement for asynchronous delegates. The newer constructs are faster, more convenient, and more flexible than the old.

To use the nongeneric Task class, call Task.Factory.StartNew, passing in a delegate of the target method:

using System.Threading.Tasks;using System.Threading;using System.Diagnostics;using System;

class Akshay{ 
    static void Run()
    {
        Console.WriteLine("Welcome to the C# corner thread pool!");
    }
    static void Main() // The Task class is in System.Threading.Tasks    {
        Task.Factory.StartNew(Run);
        Console.Read();
    }
}

Output :

Task.Factory.StartNew returns a Task object, which you can then use to monitor the task-for instance, you can wait for it to complete by calling its Wait method.

The generic Task<TResult> class is a subclass of the nongeneric Task. It lets you get a return value back from the task after it finishes executing. In the following example, we download a web page using Task<TResult>:

class Akshay{
    static void Main()
    {
        // Start the task executing:        Task<string> task = Task.Factory.StartNew<string>
        (() => DownloadString("http://www.c-sharpcorner.com/"));
        // We can do other work here and it will execute in parallel:       //RunSomeOtherMethod();        // When we need the task's return value, we query its Result property:        // If it's still executing, the current thread will now block (wait)        // until the task finishes:        string result = task.Result;
    }
    static string DownloadString(string uri)
    {
        using (var wc = new System.Net.WebClient())
            return wc.DownloadString(uri);
        Console.Read();
    }
}

Entering the Thread Pool Without TPL using ThreadPool.QueueUserWorkItem

You can't use the Task Parallel Library if you're targeting an earlier version of the .NET Framework (prior to 4.0). Instead, you must use one of the older constructs for entering the thread pool:ThreadPool.QueueUserWorkItem and asynchronous delegates.

The ThreadPool.QueueUserWorkItem method allows us to launch the execution of a function on the system thread pool. Its declaration is as follows:

ThreadPool.QueueUserWorkItem(new WaitCallback(Consume), ware);

The first parameter specifies the function that we want to execute on the pool. Its signature must match the delegate WaitCallback.

publicdelegate void WaitCallback (object state);

Again, the simplicity of C# and the dotNet framework shine through. In just a few lines of code, I've recreated a multithreaded consumer-producer application.

using System;using System.Threading;using System.Diagnostics;public class Akshay{
      public int id;
      public Akshay(int _id)
      {
          id = _id;
      }
}class Class1{
      public int QueueLength;
      public Class1()
      {
          QueueLength = 0;
      }
      public void Produce(Akshay ware)
      {
          ThreadPool.QueueUserWorkItem(
          new WaitCallback(Consume), ware);
          QueueLength++;
      }
      public void Consume(Object obj)
      {
          Console.WriteLine("Thread {0} consumes {1}",
          Thread.CurrentThread.GetHashCode(), //{0}
          ((Akshay)obj).id); //{1}
          Thread.Sleep(100);
          QueueLength--;
      }
      public static void Main(String[] args)
      {
          Class1 obj = new Class1();
          for (int i = 0; i < 100; i++)
          {
               obj.Produce(new Akshay(i));
          }
         Console.WriteLine("Thread {0}",
         Thread.CurrentThread.GetHashCode() ); //{0}
         while (obj.QueueLength != 0)
         {
               Thread.Sleep(1000);
          }
          Console.Read();
     }
}

Ouput :

Synchronization Objects

The previous code contains some rather inefficient coding when the main thread cleans up. I repeatedly test the queue length every second until the queue length reaches zero. This may mean that the process will continue executing for up to a full second after the queues are finally drained. I can't have that.

The following example uses a ManualResetEvent Event object that will signal the main thread to exit.

using System;using System.Threading;using System.Diagnostics;public class Akshay{
      private bool WaitForComplete;
      private ManualResetEvent Event;
      public int QueueLength;
      public int id;
      public Akshay(int _id)
      {
           id = _id;
      }
      public void Wait()
      {
          if (QueueLength == 0)
          {
               return;
          }    
          Event = new ManualResetEvent(false);
          WaitForComplete = true;
          Event.WaitOne();
     }
     public void Consume(Object obj)
     {
          Console.WriteLine("Thread {0} consumes {1}",
          Thread.CurrentThread.GetHashCode(), //{0}
          ((Akshay)obj).id); //{1}          Thread.Sleep(100);
          QueueLength--;
          if (WaitForComplete)
          {
               if (QueueLength == 0)
               {
                    Event.Set();
               }
          };
     }
}

Ouput :

When the consuming thread finishes consuming a ware and detects that the WaitForComplete is true, it will trigger the Event when the queue length is zero. Instead of calling the while block when it wants to exit, the main thread calls the Wait instance method. This method sets the WaitForComplete flag and waits on the Event object.

Why we need thread pooling?

Thread pooling is essential in multithreaded applications for the following reasons.

• Thread pooling improves the response time of an application as threads are already available in the thread pool waiting for their next assignment and do not need to be created from scratch.
• Thread pooling saves the CLR from the overhead of creating an entirely new thread for every short-lived task and reclaiming its resources once it dies.
• Thread pooling optimizes the thread time slices according to the current process running in the system.
• Thread pooling enables us to start several tasks without having to set the properties for each thread.
• Thread pooling enables us to pass state information as an object to the procedure arguments of the task that is being executed.
• Thread pooling can be employed to fix the maximum number of threads for processing a particular request.