Logging in CMTrace format from PowerShell

I know there are other examples of doing this out there but the one I use has a few other features including:

  • Getting the PID of the PowerShell instance running and putting that in the Thread column
  • Renaming the file to lo_ once it gets too large
  • Setting a variable to the status of the script based on the logging (used to display whether the script had an error or not)
  • Logging to the current directory the script exists in
  • Verbose Logging as an option

Here is the code:

function LogIt { param ( [Parameter(Mandatory=$true)] $message, [Parameter(Mandatory=$true)] $component, [Parameter(Mandatory=$true)] $type ) switch ($type) { 1 { $type = "Info" } 2 { $type = "Warning" } 3 { $type = "Error" } 4 { $type = "Verbose" } } if (($type -eq "Verbose") -and ($Global:Verbose)) { $toLog = "{0} `$$<{1}><{2} {3}><thread={4}>" -f ($type + ":" + $message), ($Global:ScriptName + ":" + $component), (Get-Date -Format "MM-dd-yyyy"), (Get-Date -Format "HH:mm:ss.ffffff"), $pid $toLog | Out-File -Append -Encoding UTF8 -FilePath ("filesystem::{0}" -f $Global:LogFile) Write-Host $message } elseif ($type -ne "Verbose") { $toLog = "{0} `$$<{1}><{2} {3}><thread={4}>" -f ($type + ":" + $message), ($Global:ScriptName + ":" + $component), (Get-Date -Format "MM-dd-yyyy"), (Get-Date -Format "HH:mm:ss.ffffff"), $pid $toLog | Out-File -Append -Encoding UTF8 -FilePath ("filesystem::{0}" -f $Global:LogFile) Write-Host $message } if (($type -eq 'Warning') -and ($Global:ScriptStatus -ne 'Error')) { $Global:ScriptStatus = $type } if ($type -eq 'Error') { $Global:ScriptStatus = $type } if ((Get-Item $Global:LogFile).Length/1KB -gt $Global:MaxLogSizeInKB) { $log = $Global:LogFile Remove-Item ($log.Replace(".log", ".lo_")) Rename-Item $Global:LogFile ($log.Replace(".log", ".lo_")) -Force } } function GetScriptDirectory { $invocation = (Get-Variable MyInvocation -Scope 1).Value Split-Path $invocation.MyCommand.Path } $VerboseLogging = "true" [bool]$Global:Verbose = [System.Convert]::ToBoolean($VerboseLogging) $Global:LogFile = Join-Path (GetScriptDirectory) 'LogIt.log' $Global:MaxLogSizeInKB = 10240 $Global:ScriptName = 'LogIt.ps1' $Global:ScriptStatus = 'Success' LogIt -message ("Starting Logging Example Script") -component "Main()" -type 1 LogIt -message ("Log Warning") -component "Main()" -type 2 LogIt -message ("Log Error") -component "Main()" -type 3 LogIt -message ("Log Verbose") -component "Main()" -type 4 LogIt -message ("Script Status: " + $Global:ScriptStatus) -component "Main()" -type 1 LogIt -message ("Stopping Logging Example Script") -component "Main()" -type 1

 

CMTrace Output (Download CMTrace.exe here.)

 

image

Small Basic Survival Guide – Got anything to add?

I’m creating this survival guide:

Small Basic Survival Guide

 

A Survival Guide is a list of resource links that you can go to in addition to the existing TechNet Wiki articles. To see all of the TechNet Wiki articles, see Wiki: Small Basic Portal.

Please add links to helpful additional Small Basic related resources (basically out of Wiki) you create or find on the web.

Here’s the Table of Contents for the different sections…

Table of Contents

 

So…

What’s Missing? 

Please check it out here and add what’s missing (or just link to the resource in a comment and we’ll add it): Small Basic Survival Guide

 

Thank you for your help!

   - Ninja Ed

Security Reviews: The Heuristics Zoo, Part 2/2

Introduction (Part I) 

The Heuristics Zoo, Part 1/2

Note: standard Disclaimer expressed in Part I applies here as well.

Heuristic 5: “Area Expertise” and “Penetration Testing”

These two seemingly different techniques share a lot in how they approach managing the complexity of security reviews, so I will consider them together.

“Area Expertise” is simply learning. Studying a technology long enough can make one a Subject Matter Expert (an “SME”) who knows all subtle interactions within the area and can recognize possible security issues there quickly. This approach is orthogonal to an “audit”. While “audit” spans a horizontal area across the product, an expertise would tend to occupy a deep “vertical” cut of a limited width. E.g., an expert may know how to write interfaces with static methods on them in Intermediate Language — but would have no idea about network security:

Penetration testing is a technique that seeks (to some degree) to re-create SME effect on a faster timescale. It starts with carefully choosing a sub-set of a product “surface” that appears to be softer. Then, with a help of tools and deep design dives it seeks to pass through the L1-L2 complexity levels and gain a control over a set of more complex interactions, typically at Level 3 and deeper. As many security heuristics would have patchy coverage at complexities above L2, chances are good that executing or examining that functionality would reveal new security holes.

Importantly, pentesting cannot be used to ensure large products security. As it focuses on higher order interactions, it would need at least something like O(N3) time for that. Time and budget limitations would prevent such an algorithm from scaling across the entire product. Pentesting resolves this contradiction by reducing N – in other words, but precisely aiming at a smaller part of a product to break through.

Therefore, the value of pentesing is akin to that of a geological drilling. It measures product “security” by assessing the difficulty of breaking through its L1-L2 security “crust”, and delivers samples not easily found on the surface.

 

Heuristic 6: “Learning from Attackers”

If you are an enterprise, you may have something like 104-1016 automatic security checks at your disposal to run against your product before giving it to the user. On top of that, there are 102-105 human verification questions available to apply, too. Seems like a lot? But a product with just 100 elements can have up to 1030 possible element interaction states! Even if 99.999% of them are void, the remainder still represents a space overwhelmingly huge and is beyond any technical or human capability. Within that vast space, where should we place our “tiny” 1016 security checks?

Well, the good news is that we actually don’t need to find all security holes. We need to find only those that attackers can and will (eventually) find. With that leverage, the effectiveness of the security assurance can be boosted by orders of magnitude, enabling practical and effective security defense.

What specifically could be done?

First, you can study past attacks against your product or other similar ones on the market. How are they typically attacked? What are the top 5 classes of security vulnerabilities they had faced? http://cve.mitre.org/ could be a good starting point for such an assessment.

Second, you can attend security conferences and talk to security researchers. Or, if travel budget is an issue, just read abstracts of their presentations and summaries of their talks. Even in such form the knowledge gain can help you predict how your product would be attacked in 1-3 years. By smartly prioritizing security assurance work then it could be shrank in volume (compared to “blind” planning) by a factor of 2-10 times.

Finally, if you are an online service, you can learn security attacking patterns from your logs! That way, attackers actually work for you. Any new creative idea they come up with becomes part of your tools set. For free. Isn’t it cool? Of course, this is easier to say than to do. Putting together a proper monitoring & detection solution is a daunting task. But the benefits are so great that I would definitely strive for that.

Mind the privacy though – you don’t want any sensitive or private users’ data to be exposed internally or externally in the process.

By the way, not only services can benefit from that approach. Traditional “box” products can collect crash reports and mine attacks in them as well, thus improving their security posture.

 

Heuristic 7: “Scenario Fuzzing”

A JPG file may miss up to 90% of information contained in the original bitmap image. Yet for a human viewer it represents all the essentials of the picture. Conceptually, this is achieved by abandoning a “dot” as a principal element of image construction and choosing a “wave” for that purpose.

Can we apply the same idea? Can a very large product be effectively represented with a number of elements substantially smaller than its feature count?

Personally, I’m quite positive that is possible, and possible in many ways. Here I will discuss one of them that is based on the concept of an end-user scenario.

What is a “scenario”? A scenario is a description of some meaningful user’s interaction sequence. An example would be: “launch the browser, login into your email account, read new messages, log-off and close the browser.” A product could be represented then as a set of scenarios it supports.

Why would that work?

First, scenario-based description is complete. The proof is obvious: if there is a functionality that is not activated in any legitimate user interaction, it is a dead code. Throw it away. No customer would notice that, ever.

Second, scenario representation is compact. Typically, scenarios drive product requirements, which drive the specifications, which produce the API, then the code, and thus the product itself. A single use case quoted above would already hit thousands APIs and dozens of features in a contemporary OS. A few hundred scenarios often completely describe even a large product.

In fact, for a product where scenarios “choose” features to participate randomly and independently of each other, the number of scenarios needed to execute all features grows logarithmically with respect to feature count: S = O(Log(N)). While that is probably not exactly how real products are structured, it still demonstrates the power of scenario-based product description at least for some cases.

Effectively, scenarios play a role similar to that of a “coverage graph”, providing access to a vicinity of each and every possible functionality combination in a product.

How do we use that to discover security weaknesses?

If you look carefully at security breaches you’ll see that most have valid user scenarios closely associated with them. When an attacker breaks into a system, he/she mostly interacts with it in ways that are fairly close to what a “legitimate” user would do. Those are only 1-3 unexpected “twists” that throw the system into a state of abnormal that may further lead to exploitability.

In other words, an attack often is a scenario that differs from a legitimate use by few changes. If so, can it be derived from a legit scenario by some morphing, using techniques such as random tweaks, grammar-based generators, genetic algorithms, or N-wise combinations building?

After all, if we do fuzz binary inputs, why not fuzz scenarios?

Many security reviewers do exactly that. They take a valid workflow and morph it. “Next step: FTP log-in. Well, what if I send my log-in packet twice after seeing a 100 response but right before a 200 is received? Or what if I start sending log-in packet, but will never finish it?”

Such variations may trigger completely new sequences of calls within the system and cause interactions between components that have never been in touch before… and sometimes result in an unexpected behavior.

Personally, that’s why I tend to start security reviews with a simple question of “please give me the user’s perspective”. That’s my minimal “fuzzing seed” – a valid use case that I can tweak to produce new variations.

Scenario fuzzing is intuitively easy. It scales well with respect to the scenarios count involved. It naturally considers multi-component interactions and can, in theory, discover completely unknown classes of security bugs.

But of course it has its limitations as well.

First, today it is primarily a human-driven process. I’d be very happy to see a tool that can do that job but I think our scientific understanding of natural languages is just not there yet. This tool would need to be able to take a valid English description of a scenario, tweak it, and produce a different (yet still meaningful) description as an output.

Of course, this does not have to be done in English. There are artificial formal languages for scenario description in software industry. They may offer a better starting point for this approach.

Second, it takes years of security training and a good bit of human creativity to come up with good scenario tweaks.

Third (and the most important) its’ practical application is limited by the knowledge limits of your data sources.

It is easy to ask crazy questions like “what if I do A before doing B, without doing C, while continuing to do D at the same time?” However, in many cases neither the specification, nor the product team would be confident in the answer! In fact, hitting a blank look and “we don’t know” is an indicator that the review is within the space that nobody has deeply thought through before.

But does that help? If most of these questions eventually lead to boring answers, people will quickly learn to ignore your requests. Because obtaining the answers might be difficult for them. It could take hours of examining the source code, or getting a response from SMEs, or painful debugging. Plus it takes great tenacity to make sure that email threads do not die, that people keep working on your questions and deliver consistent answers.

So it requires building great trust with people. While operating in this mode, you need to make sure you are asking “good” questions frequently enough so that people would not learn to dismiss them as “mostly useless” based on past observations.

What is the complexity of this algorithm? It’s hard to say. It really depends on the specific mechanism chosen to generate new scenarios. And it does not even have a defined stopping point, so formally speaking this is not even an “algorithm”.

Just for the sake of illustration, we can assess the complexity for one special case when:

  • Fuzzed scenarios are produced by random tweaks of legitimate scenarios, and we do t tweaks per each one
  • Scenarios use features randomly and independently of each other

For that case, the complexity of the review could be shown to be approximately O(t*Log(N)) – in other words, it’s less than linear! How’s that possible? The short answer is because in scenario fuzzing, one question executes many features, and the reviewer does not need to explicitly know them all. So he/she doesn’t have to spend O(N) time enumerating them – that has already been done by the engineering team when they designed and built the product.

 

Item 8: What about Secure Design and Layers of Abstraction?

I’ll say it upfront: neither of them is a review heuristic. They are design techniques. But as it’s simply impossible to avoid this topic while discussing software security, I’ll share my view on it.

So, in a system of N elements where each can (generally) interact with each other one, there are potentially up to 2N various interaction combinations possible. That causes security review to be exponentially complex and generally unmanageable in practice.

But what if we avoid that “any-to-any” interaction pattern and bring some structure into the system?

One way of doing that could be logical arrangement of product elements into a two-store hierarchy where:

  • The lower level elements are bucketed into groups each “reporting” to a parent element. Lower elements can freely talk to each other within the group, but cannot converse across the group boundary
  • All cross-group interactions happen through the top-level elements that talk only to each other or their “child” groups.

Here is how it may look like:

What would be the total number of checks needed to ensure the secure design of such a system? Obviously, it will require 2k checks to cover the upper level, plus k times 2N/k for the lower:

T = 2k + k*2N/k                   [3]

What is the choice of k that minimizes that number? That’s answered by demanding that ∂T/∂k = 0 and solving the resulting equation. While the precise solution is impossible to express in common functions, a good approximation for N >> 1 is easy to obtain:

k ≈ N1/2 + ¼ Ln(N)           [4]

With the corresponding number of needed security checks being then:

[5]

While that is still a huge number, it is tremendously smaller than 2N checks needed for a “disorganized” system where just everything interacts with everything. So arranging a system as a 2-layered hierarchy brings in a great reduction in complexity of the security assurance.

Can we capitalize on that and introduce more layers of abstraction, somewhat similarly to the OSI model?

The answer is positive. For a system with L such layers the total number of security checks needed (very approximately and assuming very large N) is even further reduced to:

Why not continue adding layers indefinitely? Because there is a design and implementation cost behind each next layer of abstraction. So there needs to be just enough of them to keep the system manageable – but not more. How many, exactly? If your budget of security checks is T, and there are N >> 1 atomic elements expected to be in the system, the count is obtained by inversing [6], and it is, roughly:

L ≈ Ln(N)/Ln(Ln(T))         [7]

For a product 10 Gb in size with 1 million security checks budget that evaluates to L10. Not surprisingly, it is quite close to the number of abstraction layers in the OSI model, because the dependency on both input parameters is rather weak.

In fact, a logarithm of  a logarithm of pretty much anything in our Universe could be considered a constant for most practical uses J

Now when we are done with that, let’s make a few sobering notes.

First, as a security reviewer, you rarely face a large product that you have a chance to design “properly” from scratch. More often you face systems already mostly designed, with a long history of past versions, people joining and leaving, patches, changes, and even acquisitions. Your responsibility is similar to that of a doctor, who needs to diagnose a patient and give actionable advice while fully accepting their background, past life events and known health issues.

Second, even if you get a chance to design something from the beginning, I seriously doubt that such a thing as “Secure Design” exists in practice. Sure, it would be nice to live in a world where no unexpected higher order interactions between components are possible, and all lower-order ones are documented. But I doubt that’s possible. In my opinion, there is no such thing as “completely non-interacting features”. There are only features with very low zero interaction probability. So every time you think you’ve eliminated a class of unwanted interference, a completely new one surfaces right behind it…

Functions A and B may never call each other. But if each one allocates one byte of memory dynamically, A may (potentially) eat the last memory byte precisely before B would need it. So suddenly B’s call becomes dependent on A’s execution state. Is that an effective control mechanism? Not really. But is that a dependency? Yes. And it may need to be factored into the analysis if the cost of a potential failure is represented by a nine digit figure.

Certainly, not keeping any function variables cures this problem. But even if software is (somehow?) completely removed from the risk picture, what about hardware failures? Believe it or not, some researchers have learned how to exploit results of memory corruptions caused by (among other things) random protons from the interstellar space hitting our memory chips — see http://dinaburg.org/bitsquatting.html for details.

My personal take on that is that there is always some probability of an unexpected interaction. Eliminating one class of it brings up the next one, far less probable but way more elusive and difficult to eliminate. So some surprises will always keep hiding somewhere within the realms of higher complexity interferences.

That being said, a “Secure Design” as something that tries to minimize side interactions is valuable. While it may not be perfectly achievable, it may still bring the security of a product couple levels up in terms of the complexity analysis needed to successfully attack it.

               To be continued…

Developing with Kinect for Windows v2 on a Mac

With the launch of the Kinect for Windows v2 public preview, we want to ensure that developers have access to the SDK so that you can start writing Kinect-based applications. As you may be aware, the Kinect for Windows SDK 2.0 public preview will run only on Windows 8 and Windows 8.1 64-bit systems. If you have a Windows 8 PC that meets the minimum requirements, you’re ready to go.

For our Macintosh developers, this may be bittersweet news, but we’re here to help. There are two options available for developers who have an Intel-based Mac: (1) install Windows to the Mac’s hard drive, or (2) install Windows to an external USB 3.0 drive. Many Mac users are aware of the first option, but the second is less well known.

First, you need to ensure that your hardware meets the minimum requirements for Kinect for Windows v2.

Due to the requirements for full USB 3.0 bandwidth and GPU Shader Model 5 (DirectX 11), virtualization products such as VMWare Fusion, Parallels Desktop, or Oracle VirtualBox are not supported. If you’re not sure what hardware you have, you can find out on these Apple websites:

Installing Windows on the internal hard drive of your Intel-based Macintosh

We’re going to focus on getting Windows 8.1 installed, since this is typically the stumbling block. (If you need help installing Visual Studio or other applications on Windows, you can find resources online.)

Apple has provided a great option called Boot Camp. This tool will download the drivers for Windows, set up bootable media for installation, and guide you through the partitioning process. Please refer to Apple’s website on using this option:

Alternative to installing Windows on your primary drive

Boot Camp requires Windows to be installed on your internal hard drive. This might be impractical or impossible for a variety of reasons, including lack of available free space, technical failures during setup, or personal preferences.

An alternative is to install Windows to an external drive using Windows To Go, a feature of Windows 8 and 8.1 Enterprise. (Learn more about this feature in Windows 8.1 Enterprise.)

In the section, Hardware considerations for Windows To Go, on Windows To Go: Feature Overview, you can find a list of recommended USB 3.0 drives. These drives have additional security features that you may want to review with your systems administrators, to ensure you are in compliance with your company’s security policies.


Getting started with Windows To Go

  • You will need the following to proceed:
  • Existing PC with USB 3.0 that has Windows 8/8.1 Enterprise installed (the “technician computer”)
  • USB 3.0 flash or external hard drive
  • Windows 8/8.1 Enterprise installation media (CD or ISO)
  • Windows 8/8.1 product key

You will need to log in as the administrator. Start the Windows to Go tool, press Win-Q to start the search, and enter Windows To Go:

press Win-Q to start the search, and enter "Windows To Go"

Launch the Windows To Go application from the list. From the main application window, you will see a list of the attached drives that you can use with the tool. As shown below, you may be alerted of USB 3.0 drives that are not Windows To Go certified. You can still use the drive but understand that it might not work or could have an impact on performance. If you are using a non-certified USB 3.0 drive, you will have do your own testing to ensure it meets your needs. (Note: while not officially supported by Microsoft, we have used the Western Digital My Passport Ultra 500 GB and 1 TB drives at some of our developer hackathons to get people using Macs up and running with our dev tools on Windows.)

"Choose the drive you want to use" window

Select the drive you wish to use and click Next. If you have not already done so, insert the Windows 8.1 Enterprise CD at this time. If you have the .ISO, you can double-click the icon or right-click and select Mount to use it as a virtual drive.

If you have the .ISO, you can double-click the icon or right-click and select Mount to use it as a virtual drive.

If you do not see an image in the list, click the Add search location button and browse your system to find the DVD drive or mounted CD partition:

Browse your system to find the DVD drive or mounted CD partition.

It should now appear in the list, and you can select it and click Next.

Select your Windows 8.1 image and click "Next."

If you need or wish to use BitLocker, you can enable that now. We will Skip this.

"Set a BitLocker password (optional)" screen 

The confirmation screen will summarize the selections you have made. This is your last chance to ensure that you are using the correct drive. Please avail yourself of this opportunity, as the Windows To Go installation process will reformat the drive and you will not be able to recover any data that is currently on the drive. Once you have confirmed that you are using the correct drive, click Create to continue.

"Ready to create your Windows To Go workspace" window

Once the creation step is complete, you are ready to reboot the system. But first, you’ll need to download the drivers necessary for running Windows on Macintosh hardware from the Apple support page, as, by default, Windows setup does not include these drivers.

I recommend that you create an Extras folder on your drive and copy the files you’ll need. As shown below, I downloaded and extracted the Boot Camp drivers in this folder, since this will be the first thing I’ll need after logging in for the first time.

Extracting the Boot Camp drivers from the Extras folder I created.

Disconnect the hard drive from the Windows computer and connect it to your Mac. Be sure that you are using the USB 3.0 connection if you have both USB 2 and USB 3.0 hardware ports. Once the drive is connected, boot or restart your system while holding down the option key. (Learn more about these startup key shortcuts for Intel-based Macs.)

Connect the hard drive to your Mac and restart your system while holding down the option key.

During the initial setup, you will be asked to enter your product key, enter some default settings, and create an account. If your system has to reboot at any time, repeat the previous step to ensure that you return to the USB 3.0 workspace. Once you have successfully logged in for the first time, install the Boot Camp driver and any other applications you wish to use. Then you’ll have a fully operational Windows environment you can use for your Kinect for Windows development.

Carmine Sirignano
Developer Support Escalation Engineer
Kinect for Windows

Key links

 

Clustered Column Store Index: Concurrency with INSERT Operations

Clustered Column Store: Insert Operations

As described in the blog  http://blogs.msdn.com/b/sqlserverstorageengine/archive/2014/07/27/clustered-column-store-index-concurrency-and-isolation-level.aspx , the clustered column store index has been optimized for typical DW scenario supporting nightly or trickle data load with fast query performance. Multiple inserts can load the data in parallel concurrently while DW queries are being run in read uncommitted transaction isolation level.

This blog describes locking behavior when data is inserted concurrently. For the scenarios below, we will use the following table

CREATE TABLE [dbo].[T_ACCOUNT](

       [accountkey] [int] IDENTITY(1,1) NOT NULL,

       [accountdescription] [nvarchar](50) NULL

) ON [PRIMARY]

 

            — create a CCI

CREATE CLUSTERED INDEX ACCOUNT_CI ON T_ACCOUNT (ACCOUNTKEY)

 

Insert Operations

Let us insert 1 row and see the locks taken. Note, we did not commit the transaction

begin tran

       insert into T_ACCOUNT (accountdescription ) values (‘row-1′);

 Here are the locks. Note, the new row is inserted into delta rowgroup which is organized as a btree in traditional row storage format. There is a new resource ROWGROUP in the context of CCI. The current transaction has taken IX lock on the ROWGROUP

 

Now, let us insert another row in another session as follows and look at the lock

begin tran

       insert into T_ACCOUNT (accountdescription ) values (‘row-2′);

Note, that the second transaction in session-55, also inserted the row into the same rowgroup. In other words, concurrent inserts can load the data into same rowgroup without blocking each other.

 

 In summary, the insert into CCI does not block other concurrent inserts and concurrent inserts load data into the same delta rowgorup. In the next blog, we will look into BulkLoad Operations

Thanks

Sunil Agarwal

 

SharePoint, Office365 & Yammer Nuggets of weeks 29 and 30

last nuggets before entering my 2 weeks of vacation;

  • Alternative approach of using SharePoint forms
    Slidedeck from Karsten Pohnke, ShareConf 2014 
  • New Azure templates to build a SharePoint 2013 farm out-of-the-box
    really impressing,  try here 
  • New interactive training for hybrid Office365 & SP2013  available
    official blogpost  
  • SharePoint Search Content Sources: Script to rule them all
    Max Melcher shares his script 
  • What´s new in Apps for Office?
    Office Garage has published a new video 
  • Data Collection Process for SharePoint
    This whitepaper describes data collecting processes for troubleshooting
  • SharePoint 2013 and SharePoint Online solution pack for branding and site provisioning
    Download it here
  • SharePoint Hybrid worksheets
    Download them here