Deploy a C# Stateful Service Fabric Application from Visual Studio to Linux

Deploy a C# Stateful Service Fabric application from Visual Studio to Linux

So, this was a real pain in the ass to get it working properly. But it works. Well, kind of.

First, Reliable Services are still in preview on Linux: https://github.com/Microsoft/service-fabric/issues/71

Full Linux support should come very soon (actually it should be available already according to the previous link...).

Now for the details about how to procede, here is some information to help others, because there is just nothing about that on Microsoft documentation and I literally lost 3 days trying to make it work.

1. Do use .NET Core 2.0 for your projects.

It is supported on Linux. On preview for now, but it works.

2. Do use the right RID for your projects.

As of today (April 2018), the right RID to use is ubuntu.16.04-x64.
Edit the csproj files of your Reliable Service projects and set the RID like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
    <RuntimeIdentifier>ubuntu.16.04-x64</RuntimeIdentifier>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

The fun part is, you should be able to provide multiple RIDs using the RuntimeIdentifiers parameter (with a S at the end) like that:

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
    <RuntimeIdentifiers>win7x64;ubuntu.16.04-x64</RuntimeIdentifiers>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

So you could build Windows binaries and Linux binaries at the same time.
But it simply doesn't work. When building the project from Visual Studio, I end up with the following directory only:

bin/Debug/netcoreapp2.0/

Only DLLs, no valid entry point. No win7-x64 folder, no ubuntu.16.04-x64, no nothing.
This is a bug, supposed to be fixed, but it's not (I use Visual Studio 15.6.2 all up-to-date as of today). See https://github.com/dotnet/core/issues/1039

3. You need a valid Entry Point for your service.

On Windows it's an executable file (*.exe). On Linux it's not. I ended up getting the Linux C# example and copied/pasted the entry point. https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-create-your-first-linux-application-with-csharp

So basically I now have on my ServiceManifest.xml file of each Reliable Service the following EntryPoint :

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="XXXX"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType.
         This name must match the string used in RegisterServiceType call in Program.cs. -->
    <StatefulServiceType ServiceTypeName="YYY" HasPersistedState="true" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>entryPoint.sh</Program>
      </ExeHost>
    </EntryPoint>
  </CodePackage>

entryPoint.sh is as follows:

#!/usr/bin/env bash
check_errs()
{
  # Function. Parameter 1 is the return code
  if [ "${1}" -ne "0" ]; then
    # make our script exit with the right error code.
    exit ${1}
  fi
}

DIR=`dirname $0`
echo 0x3f > /proc/self/coredump_filter
source $DIR/dotnet-include.sh
dotnet $DIR/NAME_OF_YOUR_SERVICE_DLL.dll $@
check_errs $?

dotnet-include.sh is as follows:

#!/bin/bash
. /etc/os-release
linuxDistrib=$ID
if [ $linuxDistrib = "rhel" ]; then
  source scl_source enable rh-dotnet20
  exitCode=$?
  if [ $exitCode != 0 ]; then
    echo "Failed: source scl_source enable rh-dotnet20 : ExitCode: $exitCode"
    exit $exitCode
  fi
fi

Both are inside the PackageRoot folder. I specified for both their properties so the Build Action is "Content" and the Copy to Output Directory is "Copy always".

4. Do NOT build using MSBuild !!

Yeah it is supposed to build Linux packages too, or at least it seems so, because MSBuild is able to produce the following files when you right click on your project and click "Build":

Don't trust the apparent success of the operation, it will miserably FAIL to properly execute when deployed. Some *.so files missing and other issues. MSBuild is buggy as hell and misbehaves regarding dependencies.

See for instance this bug report: https://github.com/dotnet/sdk/issues/1502
Still not fixed after almost a year...

Or https://github.com/dotnet/core/issues/977 (got this one, too).

5. Do write some PowerShell script to build the stuff by yourself.

I ended up reinventing the wheel using the following script to build my package:

# Creating binaries for service 1
cd DIRECTORY_OF_MY_SERVICE_1
dotnet publish -c Release -r ubuntu.16.04-x64

# Creating binaries for service 2
cd ..\DIRECTORY_OF_MY_SERVICE_2
dotnet publish -c Release -r ubuntu.16.04-x64

# Creating binaries for service 3
cd ..\DIRECTORY_OF_MY_SERVICE_3
dotnet publish -c Release -r ubuntu.16.04-x64

# Copying ApplicationManifest.xml
cd ..
mkdir PKG\ServiceFabricApplication
echo F|xcopy "ServiceFabricApplication\ApplicationPackageRoot\ApplicationManifest.xml" "PKG\ServiceFabricApplication\ApplicationManifest.xml" /sy

# Copying Service1 files
mkdir "PKG\ServiceFabricApplication\Service1Pkg"
mkdir "PKG\ServiceFabricApplication\Service1Pkg\Code"
xcopy "Service1\PackageRoot\*" "PKG\ServiceFabricApplication\Service1Pkg" /sy /D
xcopy "Service1\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service1Pkg\Code" /sy

# Copying Service2 files
mkdir "PKG\ServiceFabricApplication\Service2Pkg"
mkdir "PKG\ServiceFabricApplication\Service2Pkg\Code"
xcopy "Service2\PackageRoot\*" "PKG\ServiceFabricApplication\Service2Pkg" /sy /D
xcopy "Service2\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service2Pkg\Code" /sy

# Copying Service3 files
mkdir "PKG\ServiceFabricApplication\Service3Pkg"
mkdir "PKG\ServiceFabricApplication\Service3Pkg\Code"
xcopy "Service3\PackageRoot\*" "PKG\ServiceFabricApplication\Service3Pkg" /sy /D
xcopy "Service3\bin\Release\netcoreapp2.0\ubuntu.16.04-x64\publish\*" "PKG\ServiceFabricApplication\Service3Pkg\Code" /sy

# Compresses the package
Write-host "Compressing package..."
Copy-ServiceFabricApplicationPackage -ApplicationPackagePath .\PKG\ServiceFabricApplication -CompressPackage -SkipCopy

sfproj file is a Visual Studio / MSBuild related project, so you need to build everything by yourself.
The script above produces the same content as the pkg folder created by MSBuild when building your sfproj using Visual Studio. It copies everything on a PKG folder at the root of your solution.

The package structure is detailed here: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/service-fabric/service-fabric-package-apps.md

6. Now it's time to deploy!

At this point I didn't trusted Visual Studio anymore, so I built my own PowerShell script:

. .\ServiceFabricApplication\Scripts\Deploy-FabricApplication.ps1 -ApplicationPackagePath '.\PKG\ServiceFabricApplication' -PublishProfileFile '.\ServiceFabricApplication\PublishProfiles\Cloud.xml' -DeployOnly:$false -ApplicationParameter:@{} -UnregisterUnusedApplicationVersionsAfterUpgrade $false -OverrideUpgradeBehavior 'None' -OverwriteBehavior 'SameAppTypeAndVersion' -SkipPackageValidation:$false -ErrorAction Stop

It reuses the Deploy-FabricApplication.ps1 script provided by the Service Fabric project template inside the sfproj project. This script parses the Cloud.xml PublishProfile and deploys to your service fabric cluster.

So you specifies the rights values on both PublishProfiles/Cloud.xml and ApplicationParameters/Cloud.xml then execute the script.

It only works if you have the certificate used to secure the cluster installed on your machine, of course.
Do note the first dot '.' is important, because if you don't use it, you'll have the following error:

Get-ServiceFabricClusterManifest : Cluster connection instance is null

See https://stackoverflow.com/a/38104087/870604

Oh, and as there are bugs on the Service Fabric SDK too, you might want to shutdown your local SF cluster too...
https://github.com/Azure/service-fabric-issues/issues/821

7. Now it's time for another deception.

It simply doesn't work, the service crashes on startup. After searching hours inside the LinuxsyslogVer2v0 Azure Storage table (the log table for Linux, located in one of the two Azure Storage Accounts created automatically with the SF cluster), I found that Microsoft own Nuget Packages were buggy too.

Specifically, the Nuget package Microsoft.Azure.Devices doesn't work on version 1.6.0. An issue with a reference of a dll not found or whatever. I rollbacked to a previous version, namely 1.5.1, and it was fixed.

At this point I didn't had anymore energy to create another Github issue about that. Sorry MS, I'm not your QA team, I'm getting tired.

8. Build again using the first PowerShell script, deploy using the second PowerShell script, and you're done.

You've finally deployed C# Reliable Services using .NET Core 2.0 from Visual Studio (kind of, as it's buggy and I used PowerShell) on Windows to a Linux SF Cluster.

Now I still have issues with my ASP.NET Core service, but it will be a story for another day.

Conclusion: TL;DR

The whole thing is a mess. Bugs everywhere. In the SDK, in the tools, in some of Microsoft Nuget Packages. Awful experience. But it is supported (in preview for now) and you can make it work. Hope this post will help...

Deploying additional DLLs with service with ServiceFabric via Visual Studio

Check out my response here: Service Fabric include additional files. You can accomplish this in one of two ways: specify the files as Content files of your service project, or manually copy the files to the service package in a post-Package MSBuild step from the application project.

Related Topics