IP restricting Windows Azure Sql Database with the Service Management API

One common request or annoyance from customers is the fact that Windows Azure Sql Database (WASD) supports two modes of protection with its firewall only.

1. External IP addresses that are used to manage databases through SMSS or some other tool

2. An “any” rule with an IP range of 0.0.0.0 – 0.0.0.0 which denotes that any Windows Azure virtual machine can access the Sql database server

Its not that clear cut but lets have a short lesson on the latter is achieved … Each role instance has an IP address of 10.x.x.x and traffic can be routed between roles on internal endpoints through this network interface. An external service like Windows Azure Sql Database has no knowledge of this since it’s not part of a virtual network. This means that when an application hosted on a virtual machine connects to the WASD the firewall will pick up the public address of the cloud service.

Try this out by removing all rules and connect to a database in a sample web application and you’ll see an IP address that is referenced within a public range here:

http://www.microsoft.com/en-us/download/details.aspx?id=29840

That being said a better way of using the firewall would be to avoid the “any” rule altogether and lock down the WASD to the specific IP address of the cloud service.

Fluent Management was conceptualised to create deployment workflows in Windows Azure and one that I like quite a lot is the idea of building an agent or a monitoring application of some kind which will monitor a cloud service deployment. If the address changes (infinite lease unlikely but possible depending on the circumstances) then the agent will pick up the fact that the IP has changed blow away the firewall rules and create a new set.

I set out to do just this and this is what my code looks like:

It’s a new SqlDatabaseClient class which now has that single method to be able to take a cloud service name and look at either the staging or production slot and optionally blow away all other rules or append the new IP to an existing set of rules.

There are several ways to do this using Fluent Management. One way could be coded as follows:

This will use the Linq provider to determine the current public IP address of the cloud service and it can be checked against the stored previous one(s). This can also be used in conjunction with the fluent API deployment for cloud services and WASD. More can be read about this on the wiki.

The latest version of fluent management is 0.4.8.5 which will be packaged and I’ll add to nuget in the next couple of days.

Creating and Deploying a Windows Azure Website in C# from Fluent Management

I’m trying to get a lot more organised these days with Fluent Management. Recently I’ve wanted to be able to create Windows Azure Websites in C# for a multitude of reasons so I’ve set about reversing the websites Service Management protocol. This is at a very early stage at the moment but I’ve made significant progress so can do the normal things like create, delete, stop and restart a website. You can also create a Website with a local repo and one with a Github repo.

I’ve had to write my own Github client since I’ve needed to use Basic Authentication where Azure gets an Auth token so currently it takes plain text credentials. In the future I plan to allow an auth token credential (if you want to set up the auth token in advance with Github). Since there is no user context with the library it has to be done this way. One other thing I’ve enabled and tested is scalability. From Fluent you can adjust the number of instances of your website in code which was really my initial desire.

Currently it’s not atomic like the Website (Kudu) implementation so it leaves the Github repo in an inconsistent state and doesn’t remove the service hooks (these are callback urls which are invoked when a push has occurred) but I plan to rectify this in future versions. It’s only a minor irritation anyway.

In order to create a basic website from Github now you can use the following code which will create a reserved small site with 3 instances and add an AppSettings, a ConnectionString and some logging site config.

For those that are new to Fluent Management you can quick start on the wiki @ http://fluentmanagement.elastacloud.com, download and fork the source from Github or download the package and get going from nuget.

The latest nuget package will be 0.4.5.5 and will be available in the next few hours (by midnight UK time:-) ). It’s worth noting that the dependency on nlog and SSMS has been removed but one has been added on the Storage Services SDK until I get the time to publish something which will support block blobs and block lists.

In the next iteration of this I plan to allow log downloads and add BitBucket and DropBox as available deployment repos. I haven’t as yet been able to automate the deployment from Github when you first create the site but I’m sure I’ll figure it out this week – it seems to be responding to the service hook when I use a test from Github but not programmatically. For the time being if you use this you’ll have to do a sync from the portal.

Anyway have fun!

Why Big Data should matter to developers

I’ve been really busy for the last month. It’s not like me to discard the blog but I’ve been so constrained for time. Things are getting easier now and I plan to blog about some of the things that I’ve been doing and problems I’ve solved that might be helpful for others.

Myself and @andybareweb are now spending most of our work life on Big Data problems (Andy especially has really pushed HDInsight usage into an attainable developer goal). Every week I read some nuance from some slick technical journalist about Big Data and “buzz” and “trend”. Some in the SQL community are awash with criticism of this new phenomenon saying that it is nothing new.

I wanted to voice my thoughts, as it has been nearly a year since I started using Hadoop and three months since I started using Spark and Scala to solve some real problems. The first thing to say is that Big Data is really cool for developers. Yes it’s nothing new but it is also is. Last week I decomposed a problem into chunks over a dataset of 6 TB so I could “machine learn” and build a recommendation engine of the back of said data. I would have never dreamed of doing that one year ago, known how or cared! That would always have been someone else’s problem but now its just part of building an application.

Now enter the cloud. That dataset could have been 600TB for all I care. For the first time I have the same resources at my disposal as companies like Netflix who store all preference data about their users and have built a model which predicts with a great deal of accuracy what they would like to see next. I can scale resources to the time I want to complete a problem rather than give an impromptu calculated based on the resources available. This is something I’ve wanted to see for the last two years as we moved forward with distributed computing paradigms. That is, the idea of scale applied to the cloud by everyone. HPC is a great solution but turned out to be fairly niche. Many people decided to roll their own and as a result there were a lot of complete abortions built by those that didn’t understand the implications of computing at scale.

One of the things that excite me about the preview release of HDInsight is that it is a one size fits all solution. Microsoft has thought about the scale unit and abstracted you from the guts of the infrastructure implementation. Service such as Templeton has now been enabled to all remote execution of jobs and to get feedback from Hadoop. In this way you just deal with an “HDInsight cluster” as opposed to a virtual machine. Whilst this is not new in its outlook other as Windows Azure services follow the same pattern it’s a very novel way to package a whole heap of useful things. HDInsight comes complete with a “machine learning” tool, a graph API, a workflow client and of course a popular map-reduce engine. One or two of the things are accessible to .NET developers now whilst others entail that you have a working knowledge of Java and Eclipse. It doesn’t matter though. If there is a will to learn this, then you just fill that gap!

So, to my mind, Big Data on Windows Azure is not buzz. It is just another tool in the developer arsenal to problem solve. The only difference this time is that we have access data and also to scale. If you can write C# (or F#) then give it a go.

We’ll be running our first hackathon http://www.bigdatahackathon.com sponsored mainly by Microsoft. We’ve almost got a full house so register quickly. It’s on the weekend of 13/14th April in London. There are some great competition prizes. We’re running this in conjunction with Data Science London. This is an important fact because the cloud developers from our community will get the opportunity to work in teams with data scientists and understand how modelling and tooling can be applied to datasets in the cloud. It will be a fantastic learning experience for anyone new in this area.

Enabling Remote Desktop on Ubuntu in Windows Azure

This is a weird post.

I decided to write this simply because there should be a little parity in the way that we manage Windows and Linux. Everything I do now with Linux I through the Terminal but with Windows via Remote Desktop. I decided to enable RD for my Ubuntu instance running Spark. I don’t actually use it but it’s nice to to have the option!

To start with install an Ubuntu desktop. The default is unity but some of you may prefer Gnome.

apt-get install ubuntu-desktop

Then you need to install an RD server which you can do like so:

apt-get install xrdp

When this is done you need to be able to configure the Windows Azure firewall. So you can see that the xrdp port uses the same port default as our RDP port on windows.

Firewall ports

Firewall ports

Easiest thing to do after that is to download Bitvise Tunnelier which I’ve discussed in a previous post.

After you’ve SSH’d in you should be able to see the Remote Desktop menu like so.

Remote Desktop menu

Remote Desktop menu

You also need to make sure that the right computer name is in the remote desktop configuration.

Remote Desktop Configuration

Remote Desktop Configuration

Click on the RD menu link and voila!

The windows remote desktop client should present something like this (the refresh sucks but it works okay):

remote desktop client

remote desktop client

Enjoy etc.

 

Deploying a Linux Web Farm on Windows Azure in 30 mins

Well, this week I organised our first open source conference at Microsoft’s offices. Special thanks to @richorama and @robblackwell for providing some great content and also to @simonevans. The day went off smoothly despite the reliance on projects called snowylondon, hailinglondon, raininglondon etc. @andybareweb started the trend but all other speaker (myself included) flogged it to death.

I did a demo which I’d like to share because I think it’s impacting for non-Windows users and demonstrates how easy it is to set up a highly available web farm. I’ve used Ubuntu but this can equally apply to all flavours of Linux.

To start with you’ll need to download the CLI from http://go.microsoft.com/fwlink/p?LinkID=275464. I’ve referenced the CLI in my blog posts but it’s basically a cross platform command line interface built in node.js which allows us to issue service management commands.

My demo was specifically going to be about deploying Ruby and Rails but it turned out there were no Ruby developers in the room so we’ll save that for another day.

Before following the setup instructions here you’ll need to import your credentials to the CLI. This is well documented on windowsazure.com.

The following will create a cloud service which will be available on http://rubyrubyruby.cloudapp.net in North Europe. It’s important to add the –location flag in the first instance since this will create a new cloud service if one does not exist already. In this case the vm will be small and the image used will be Ubuntu.I’ll be using SSH on port 64562 with an initial username and password as described. Our vm will be called ossazure1.

azure vm create –location “North Europe” –vm-size small –vm-name ossazure1 –ssh 64562 rubyrubyruby b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_1-LTS-amd64-server-20121218-en-us-30GB azurecoder P@ssword123

We should get the following feedback from the CLI:

info: Executing command vm create
+ Looking up image
+ Looking up cloud service
+ Creating cloud service
+ Retrieving storage accounts
+ Creating VM
info: vm create command OK

Once this is done we can issue a second command to create another vm instance in the same cloud service like so.

azure vm create –connect –vm-size small –vm-name ossazure2 –ssh 64563 rubyrubyruby b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_1-LTS-amd64-server-20121218-en-us-30GB azurecoder P@ssword123

We’re calling this one ossazure2. As you can see the SSH port is different. Since the cloud service is effectively “bound” to a single soft network interface it’s important to distinguish ports as the CLI will generate a conflict if we use the same SSH port. Since the cloud service now exists we no longer have to use the –location flag.

We can see here that the endpoint was create and has a different public and private port.

endpoint sshendpoint ssh

endpoint ssh

In order to create resilience I want to ensure that a power outage in one area of the data centre doesn’t affect both VMs. This is done through Availability Sets. We’ll do this part through the portal because I like the visual feedback it gives you.

First create a new availability set and save this to ossazure1.

Creating an availability set

Creating an availability set

It may take a while because things are being shifted around in Windows Azure to ensure that the machines are on different switches and different areas of the data centre.

Availability set ossazure1

Availability set ossazure1

You can then add ossazure2 to this and you should see the visual feedback below showing the members of the availability set. I can carry this on ad-infinitum but we’ll stop at two vms for brevity.

Availability set 2 vms

Availability set 2 vms

availability set both vms

Now we’ve got a resilient system we want to create a Load Balancer to allow traffic to be evenly distributed between the two instances. We can do this through the CLI like so:

azure vm endpoint create –lb-set-name tom ossazure1 80 80
azure vm endpoint create –lb-set-name tom ossazure2 80 80

Giving the load balancer a name is not optional but you don’t have to call it Tom! Miraculously we get the following setup for us on a new web endpoint.

Load balancing on the web port

Load balancing on the web port

Notice the load balance YES in the image. This enables me to define the same port for both vms. By default the load balancer is set to round robin between vms but you can easily setup a probe through the CLI (we can do this in another post) which will allow the load balancer to check for the health of a web page and if it isn’t up not serve from that vm.

In order to complete this we simply use an SSH client with the above credentials and install apache. The following script should help you get started:

#get updates
sudo apt-get update
sudo apt-get upgrade –show-upgraded
#install apache
sudo apt-get install apache2 apache2-doc apache2-utils
#start apache
apachectl start

It will have to be done on both machines.

To check whether this works change the text on one of the Apache index.html pages.

cd /var/www
chmod a+rw index.html
sudo vi index.html

Make the changes and then check your service – in my case http://rubyrubyruby.cloudapp.net – it will sort of round robin the requests (it’s a little sticky) but it works.

Happy trails!

An architectural pattern for sending notifications from cloud services via mobile services

Mobile services is a great boon for developers like myself that have mainly had contact with web or windows only applications in the loosest sense. What I’ve found myself doing is coveting parts of mobile service applications but not the whole. Whilst the authentication piece and data management are indeed incredibly useful if you’re developing a greenfield solution, in the event that you have an existing investment in an ORM such as Entity Framework or nHibernate it’s not much use.

Having authentication providers means that you can avoid implementation of something like DotNetOpenAuth in your application. The truth is however, that mobile services is what it says it’s a service for mobile developers so in effect authentication would be used directly from a mobile device as would all of the services it provides. When you have an existing application which acts as a relying party you would want to use this to

The case I’m considering though is when you have a complex application which requires reuse of an existing deployment. Well generally you could deploy the application to a cloud service and there would be a number of ways you could leverage some of the services in mobile services.

Let’s say in my application I already have a Users table in the dbo namespace. I would be able to use simple Sql statements against this to get my user information such as:

SELECT * FROM DBO.USERS

If we go back to my original post on notifications here let’s say that I wanted to create a Channels table as in the post. I could use the same database as my users table by configuring mobile services to use this table as per this image:

Mobile Scheduler

Mobile Scheduler

How I create the Channels table is relevant here because I want mobile services to be aware of it but all of the inserts will be occurring through my cloud service application.

Mobile Architecture

Mobile Architecture

If EF I might build my DbContext in the following where the schema is the name of the mobile service. The Channels model could have a foreign key constraint which would be the UserId from the dbo.Users table and every time I created a new user I could add the Channel through EF. Doing it this way would avoid any code executing on the Insert as we wouldn’t have used mobile service’s REST API to do the insert.

Let’s assume that we have a table which captures the notifications that an Administrator wants to send to the users. The table could be called dbo.Notifications. The mobile services application, like the dbo.Users table would not be aware of this.

So we could create a new Scheduled job and pickup the notifications every few minutes, send to all of the ChannelUri’s in the Channels table and then delete the rows from the Notifications table.

In order to select this using and kicked off by a scheduled job (you can read about scheduled jobs here) we could write a node function set like so:

I wanted to present a very simple reuse pattern here which leverages a key service provided by mobile services which can fit into your existing application. Microsoft has provided alternatives to this with a new notification hub using the Service Bus but the pattern described here is a viable low cost alternative. It’s important to note that the Channels table does not need to be in the mobile services schema in this example – the benefit of doing so however is that you can make a request of some kind – an update or delete of the channel, for example and send a message to the user’s device using a script.

30 mins to a Ruby on Rails application with the Windows Azure vmdepot (Part 1)

Note: If you like this post, the UK Windows Azure Users Group will be running a free conference day in London on the 11th February which you can register for here: http://lanyrd.com/2013/open-source-on-azure/. Windows Azure is a great cloud platform to kickstart development with non-Microsoft languages and frameworks. If you haven’t used it before we’ll show you how to get up and running, what it can do for you and how you can get your hands on some free time to have a play.

I’ve been working quite a bit on a project with Ruby on Rails and Scala. It’s an open source project and I’ll be blogging about it and writing a paper shortly but in advance of that I thought I’d write a two part article on how you can get a Ruby on Rails test environment up and running within a short space of time. In this post I’ll cover how you would do this with a developer image that Microsoft and their partners are making available which contains pretty much everything you need to get up and running.

Everything in this post is based on Infrastructure as a Service (IaaS). In the next post I’ll look at how we can use PaaS with Windows Azure Sql Database and a Worker Role to build an automatically scalable Ruby applications.

Everything in this post took me in the region of 30 mins to setup. I’m fairly new to Ruby and don’t use it at all in my day to day coding. I have fallen in love with its simplicity and sheer community support (with a great ecosystem to match). It’s evident to me now that some really good parts of ASP.NET MVC were copied from Ruby on Rails. It makes me happy that Microsoft has learned to watch community development and respond to how people want to use tools in the real world.

To begin with we’ll navigate to http://vmdepot.msopentech.com/List/Index and in the search box type in devpack.

Press the triangle image marked deployment script. Follow the instructions and copy the resulting text. The output should look this and you can see I’m deploying this to North Europe and using a domain name called rubytime and a username and password of azurecoder and MyRubySuccess9000 in this case.

> azure vm create rubytime -o vmdepot-77-1-16 -l “North Europe” azurecoder MyRubySuccess9000 –ssh

If you haven’t already downloaded the CLI then do it’s amazing and supported on Windows, Linux and MacOSX. It’s written in node.js so will install the node runtime locally.

http://www.windowsazure.com/en-us/develop/nodejs/how-to-guides/command-line-tools/

 

You should of course also have an azure subscription and follow the documentation above to setup an account. You can get free trials for 90 days which will allow you run a small instance.

I downloaded Bitwise Tunnelier to use SSH from my Windows 8 laptop. Once you’ve followed the above instructions and created the VM you can go about setting up a basic Rails site. The devpack contains a wealth of runtimes and tools including PHP, Ruby, Python, Java and various frameworks. Great for building and testing your code on. It also contains mysql, postgress sql, apache and nginx as well as support for git and svn.

To get up and running with a new Rails application enter the following at the SSH prompt:

> cd stack
> sudo ctlscript.sh stop apache

The above will run a pre-defined script to stop apache. We’re going to test with the Rails server first.

Now we’ll create the Rails application called test.

> sudo rails create test

The above will generate an application called test with the default Rails skeleton. We should generate a default controller and view called index and create a default model.

> rails generate test controller index
> rails generate test model

We should navigate to the test directory and edit the index view adding some text like “Welcome to Ruby on Rails on Windows Azure!”.

Now we can navigate to the demo/config directory.

> cd demo/config

And edit the route.rb file with the following to point to our new site index view.

root :to => ‘test#index’

We then rename the public index file to index.old otherwise the rails server will display the default.

rename public/index.html to index.old

We’re now ready to run the Rails server.

> sudo rails s -p80

Before being able to see our masterpiece on the public internet we’ll have to add an endpoint through the management portal for port 80. Navigate to your Windows Azure VM instance on the management portal and add an Endpoint entry called HTTP, set the protocol to TCP and add port 80.

Okay, so now that we’ve got all of this working we want to extend our new application to be able to use a database. As luck would have it the Bitnami image comes with MySql already installed, configured and running and allows us to login to PhpMyAdmin which will enable us to add some test data.

In order to facilitate access we’ll stop the Rails server using the terminal and run the script in the stack directory using:

> sudo ctlscript.sh start apache

And then setup a rule in our SSH client so that we can port forward.

Setting up SSH port forwarding

Setting up SSH port forwarding

We can then request http://localhost:8888/phpmyadmin/ so that we can use PhpMyAdmin. Logging into PhpMyAdmin can be done using root as a username and bitnami as a password. Not very secure but we can always change this later on (also we haven’t exposed the MySql service to the world so nobody without the SSH login details will have access at this stage).

MySql Admin homescreen

MySql Admin homescreen

To test this out I’ve created a simple Users table with four columns: UserId, Username, Firstname and Surname.

Creating the table

Creating the table

We should create a couple of test users like so.

Adding table rows

Adding table rows

Now that we’ve completed building a table with a couple of users we’re primed to be able to use ActiveRecord and create a Rails application to display our data.

We’ll now update our database.yml file which resides in the db directory off of the Rails application root. It should look something like this:

 

test:
  adapter: mysql2
  database: demo
  host: 127.0.0.1
  port: 3306
  username: root
  password: bitnami

Okay. All good. We’ve proved we can create a database and add rows! We’re going to junk all of this though because Ruby does everything much better and quicker than we can. You can see from the trace below that if we use rails to scaffold everything it will create everything that we need to build a complex user model including a variety of CRUD functions in a custom controller for the model, a migration to allow us to add the new model to the MySql DB.

azurecoder@rubytime:~/demo$ rails generate scaffold User user_id:int username:string firstname:string surname:string
invoke active_record
create db/migrate/20130127121055_create_users.rb
create app/models/user.rb
invoke test_unit
create test/unit/user_test.rb
create test/fixtures/users.yml
invoke resource_route
route resources :users
invoke scaffold_controller
create app/controllers/users_controller.rb
invoke erb
create app/views/users
create app/views/users/index.html.erb
create app/views/users/edit.html.erb
create app/views/users/show.html.erb
create app/views/users/new.html.erb
create app/views/users/_form.html.erb
invoke test_unit
create test/functional/users_controller_test.rb
invoke helper
create app/helpers/users_helper.rb
invoke test_unit
create test/unit/helpers/users_helper_test.rb
invoke assets
invoke coffee
create app/assets/javascripts/users.js.coffee
invoke scss
create app/assets/stylesheets/users.css.scss
invoke scss
create app/assets/stylesheets/scaffolds.css.scss

Now that we’ve built the scaffold we’re ready to test our application and see whether we can insert and read users.

Don’t do this at home but for brevity I decided to enable all routes in the routes.rb file. I’m having a lazy moment and I’m using Windows Azure so this will be down within the next few hours but every Rails site I looked at including the comment code in the routes says this is a bad idea for your application.

match ‘:controller(/:action(/:id))(.:format)’ 

Anyway, to create a new database I just navigate back to the root and issued the following:

> rake db:create

 Now that this has been done it’s easy to add stuff to the new Db through the views that have been created for us (first run the server again using the sudo rails s -p80 command):

Adding a user to the database

Adding a user to the database

The upshot is that I can now add a user and check to see whether the user has been added.

Listing users

Listing users

There you go. Even for a relative novice like me this is a great developer experience without the need to install anything.

 

Using Windows Azure Active Directory (WAAD): Part 3 (Custom Web Principals)

Well, here’s the third part of the article on WAAD that I promised. Using the Graph and integrating this with the web is actually quite easy. You should be able to make use the wrapper of I wrote up in the second article here and build a new web principal which can be used to authenticate and authorise your Windows Azure Active Directory users.

Application Architecture

Application Architecture

We looked here at building a library which could abstract the use of Microsoft’s AAL library and provide the idea of a generic user with a set of properties and methods that can be used in any application. In order to consume this in a web layer though we need to create a web principal which can be used to pull back all of the details of the user through the graph.

The WAAD currently has several limitations when compared to a normal Active Directory. When you think about it this is obvious as the user only needs certain available claims when used in a web context. The problem we have though is that some systems use AD to store metadata about the user which is not then possible in WAAD. WAAD does not have the ability to have extensible metadata additions for the user; it doesn’t actually need to and the point is not to emulate Active Directory since that exists already!

Therefore, any web principal we build will have to implement a hybrid between a database repository of some type and the basic details from the WAAD.

The other problem is the use of roles. In WAAD terminology and for our purposes a security group is a role which means that we have to abstract this within our custom web principal.

In general we’ll need a couple of things. We’ll need to wire up single sign in through ACS which you can read about here. We’ll also need to create a custom principal from a ClaimsPrincipal which is returned through the ASP.NET runtime. I’ve implemented an interface with the custom principal just so that it is testable.

You can see here that the underlying implementation will ensure that the user has a database profile which will be created lazily. This is a great way to create an onboard users and slowly build up web profiles for them despite their affinity to WAAD. The Graph isn’t fully functional currently and this will be a much faster way to do things. You have already ensure that username or UPN is unique so indexing can occur on that. Something to note is the use of differentiation between IsAdministrator and IsSuperAdministrator where a SuperAdministrator can own the application by virtue of security group membership and the Administrator which is a tenant can be a member of an organisational security group and also the Administrator security group so that the set determines the underlying management role.

Note the implementation of IsInRole on the principal just uses the IsInGroup method on the AD user so we’ve done a transposition in terminology. A good way to evolve this is to use security groups as well for metadata. When managing your own AD and creating multi-tenant applications WAAD can be used in a way which will allow identifiers such as corporation names as being synonymous with security group names to supplement the lack of available metadata at source. It’s a good and simple pattern as well.

When we implement the IIdentity interface for our custom identity we can simply use the claims to return information about the user. It’s worth noting that the claims returned for the user are not wired correctly with the ASP.NET runtime. There’s a couple of approaches to this and to avoid a null value for Identity.Name. The first is to do what you see below and pull out the Upn value (Unique Principal Name) from the claims.

The second is to update the web.config. The default implementation is to look for the name. For a web principal the name or email claim will be used as it assumed to be unique. In this case the email isn’t present from our ACS output claims so we have to supplement this or change the name claim that we look for in the web.config which will automatically update the Identity.Name.

Lastly, we need to consider how we create the principal so we’ll focus on the global.asax and the ASP.NET pipeline and create the custom principal from the incoming claims principal.

In order to use this declaratively simply decorate a Controller method like so.

Hopefully you can see from this series that wiring up ClaimsPrincipal authenticated via WAAD to be used in a web application is as easy as RaspberryPi.

 

Building Service Management API requests for Windows Azure Mobile Services in Fluent Management

Over the last few days I’ve been having some fun with mobile services specifically trying to reverse the CLI tools by
tracking their HTTP requests-responses. Somebody recently asked me if we (as a company) had the capability to add a Mobile Service deployment to our orchestration for Fluent Management and since I’ve been working with mobile services for the last few months I figured I might as well give it a go and decided to try and work out the Service Management requests I need. Everything I write here is specific to the mobile services addition to Azure Fluent Management. To get started with Fluent Management have a play with the latest version on nuget or check out the code on Github. For a quickstart guide you can look at the wiki here.

In order to facilitate an experience akin to the new CLI tools I wanted to copy the parts which would make a good stateful experience and allow for a full deployment of a mobile service. Give the client in Azure Management a test run but weary of deploying and using in a production system just yet until the v1 of the API is published by Microsoft. Until then the CLI is a great resource and pretty much does everything you need for a full scale deployment of Mobile Services.

Currrently there are two things I’ve left out in my implementation:

  • Apns (Apple Push Notification Service)
  • Scheduled Jobs

The former I’ll add in the next few weeks as I have a need for it going forward. The latter I’ll have to take a closer look at and maybe see if I can get a heads up from the mobile services and/or the CLI team. Remember, this is all reverse-engineered and there is no published spec for this API. Given what I’ve seen of it I’m sure it will change considerably before being released as well. For my part I’ll keep Fluent Management up-to-date and hopefully there will be some backward compatibility with operations. There’s also some feedback I’ll give to Microsoft since I’ve come across some issues in attempting to do this.

It’s important to say that I haven’t yet implemented this as transactional component of Fluent Management but it is a stateful client called MobileServicesClient which is fairly feature rich. In the next month I will as I have to ensure that Cloud Services, Storage and Db’s are created transactionally with the Mobile Service so watch this space. For the time being the client is pretty much fully functional apart from the above so should suit most scenarios where the CLI is not applicable.

The current client contains the following features:

  • Create a new mobile service with a new linked Db
  • Get the context of an existing mobile service
  • Return all mobile service settings including identity and push notifications
  • Return all mobile service tables with associated permissions
  • Add new tables
  • Add new scripts and update table permissions
  • Update all properties (including dynamic schema)
  • Regenerate application and master keys
  • Restart the mobile service
  • Return the top 200 logs and show incremental state filtered by date

The following shows what you can do with an existing mobile service. Since Fluent Management has had a rehaul I’ve built new mock tests with @andybareweb but have foregone all of the functional tests. For now I’ve decided to build a fully functional console application as a test harness which will leverage fluent management. It’s not that I’m trying to write another set of powershell CmdLets or CLI tools with this because Fluent Management was simply released to provide a fluent interface to deployments and to build orchestration patterns atomically for deployment of all dependent Azure  services. It is, however, a good way to test the fluent interface where we cannot test through mocks.

And you can also just build a new mobile service like so:

As you can see building a new mobile service is automatically stateful so you can just keep the MobileServicesClient context and do any available operations on it.

Hope you enjoy this. For me it was a couple of days of intense effort but I’m very happy with the results.

Using Windows Azure Active Directory (WAAD): Part 2 (Graph API)

In the previous post we looked at how to setup a Windows Azure Active Directory. In this post we’ll go about consuming it. In looking into this I found a disconnect with being able to hit the ground running with building a user in WAAD and building a principal in my web application. My aim is simply to be able to use metadata from my Active Directory users so that I can effectively have a single web principal whether the user has logged in via Active Directory or via another identity provider such as Facebook.

Firstly, the project is parked on github with a test project that you can begin using straight away. You can view this here. I’ve uploaded a nuget package as well which you can get started on here.

In the last post we looked at building a new WAAD tenant so that we can set this up as an Identity Provider in Access Control Services and then use this to login to a web application via Single Sign In. WAAD is still pretty virgin so there are a few gotchas when you go through that process. When that is done though you’ll want to do something with the AD user and currently there is a disconnect between getting this going out of the box. In this post we’ll look at the Graph API which is a REST-based API which exposes Active Directory objects for a particular WAAD tenant.

In order to get started let’s take a look at a great test application at https://graphexplorer.cloudapp.net where we can peruse the graph before we begin to write code. So to start with navigate to the site.

In order to begin enter into the resource textbox:

http://graph.windows.net/mydomain.onmicrosoft.com (replace the mydomain with your domain – unless you’ve managed to bag mydomain!)

You should see the following:

viewing the graph

When you click on any one of the links you’ll be presented with the following screen which will allow you to login to the AD.

login to the graph

When you enter your credentials (AppPrincipalId and SymmetricKey from the previous post) you added earlier you’ll be greeted with the following error because the ServicePrincipal we created in the previous post is not authorised for the graph.

{
“Status Code” : “Unauthorized”,
“Description” : “The remote server returned an error: (401) Unauthorized.”,
“Response” : “{“error”:{“code”:”Authentication_Unauthorized”,”message”:{“lang”:”en”,”value”:”Unauthorized request.”}}}”
}

In order to authorise the new ServicePrincipal enter the following in the Msol CmdLets and try again.

You should be able to see the graph now like below.

querying the graph explorer

You’ll see that the response is clickable and you can build simple graph queries to enable you to return various users and the groups that they are members of. Everything in the graph has a unique ObjectId (where xxx is the Guid ObjectId in the query below) and you’ll be able to see from the trace that the graph uses these and filters on them to move between nodes and node sets. One example to get all of the roles and groups for a particular user is a query like this. You’ll see that everything returned is of type ReferencedObject.

https://graph.windows.net/adelastacloud.onmicrosoft.com/Users(‘User_xxx’)/MemberOf

By now you should be clicking around the graph all over the place!

If we turn our attentions to code for now and look at how this all hangs together. In the test project this is what I want to accomplish which should be fairly self-explanatory.

The code looks to check the following for a particular user:

  1. That the user exists
  2. The the user belongs to one or more security groups
  3. Gets some personal information about the user
  4. Gets the user’s thumbnail image if it exists

The user is identified by the UPN (Unique Principal Name) which will be something like myname@mydomain.onmicrosoft.com. Certainly a better reference than the ObjectId! One thing that should strike you immediately is the use of security groups instead of roles. With a traditional Active Directory there are two things you can do which can be helpful within an application.

  1. Add additional metadata for the user
  2. Add roles and put the user into these roles

The latter is used throughout authorisation. The roles for the WAAD are, however, not usable (and configurable through the GUI) in the same way so we can create “security groups” instead and add the user to them. From now on we will treat these groups like roles.

From the above code it can be seen that the DirectoryDataService connects to the graph initially and uses an event to intercept the request so that the correct auth headers can be added. The following is added to the request:

  1. The authorization header from shared secrets (the ServicePrincipal’s symmetric key)
  2. The API contract version for the graph
  3. If this is a continuation of an existing session then the session key is added
  4. A unique id for the request

You can see from the creation of the authorisation headers that there is a reserved principal guid for the WAAD ServicePrincipal which is used like so 00000002-0000-0000-c000-000000000000/graph.windows.net@mydomain.onmicrosoft.com. This is called the ServiceRealm and a combination of this and the AppPrincipalId and SymmetricKey is used to build the header. At this time I have a dependency on AAL which abstracts this but you can get into the guts of the protocols and build a JWT (Json Web Token). Since AAL is a rich library with a lot of features (which I’m not using – such as GUI windows etc.) I may write my own JWT parser in the next iteration. Anyway, the majority of the connection code in ActiveDirectoryRequest.cs is cobbled together from a Microsoft sample.

The simplification comes from the ActiveDirectoryUser.cs file which contains the code which abstracts a user in the WAAD. In the first we need to check whether the user exists in the WAAD using:

This will perform an OData query and use the $filter on the UserPrincipalName which should be the email address for Office365 or the fully qualified name for a new Active Directory tenant. If this returns with a value we have access to all entered metadata about the user.

The Groups property returns the groups that the user is a memberof by using the MemberOf syntax we looked at with GraphExplorer. Since this query returns all of the memberships including Roles which we’re not interested in we simply return the group names. You’ll see from the WAAD portal that all we have is a name and a description for the group so we’ll just discard the description from now. The IsInGroup method therefore checks only whether the user belongs to the group which is already in the memberships list.

Apart from all the other supplementary user properties the only thing worth diverting attention to is the thumbnail part of the code. One of the requirements I had was to display the thumbnail image from the WAAD so I exposed a thumbnail property. There is a bit of jiggory-pokery going on in the ActiveDirectoryRequest.cs GetStreamData method if you look through the code but otherwise it is fairly straightforward.

Anyway, go ahead and feel free to adapt and change. I’ll be doing another post in the next week or so to give a bit of focus to this and show how you would consume this in a web principal.

Have fun and happy holidays!