ForEach and Where magic methods

ForEach and Where are two frequently used concepts that have been available in PowerShell since version 1 came out in 2006.  ForEach has been available as both a statement and a cmdlet (ForEach-Object), allowing you to iterate through a collection of objects and take some action once for each object in that collection.  Where has been available as a cmdlet (Where-Object), allowing you to filter out items in a collection that don’t pass some condition, a condition which may be evaluated using either the properties on objects in the collection or the objects themselves that belong to the collection.  The ForEach ability to take action on items in a collection and the Where ability to filter a collection are features that are highly useful and are included in one form or another in the logic behind many PowerShell scripts, commands and modules, regardless of what version of PowerShell is being used.  In fact, they are so heavily used that they have been a focus area for improvement of performance, functionality and syntax in PowerShell versions 3.0 and 4.0.

With the release of Windows PowerShell 4.0, two new “magic” methods were introduced for collection types that provide a new syntax for accessing ForEach and Where capabilities in Windows PowerShell.  These methods are aptly named ForEach and Where. I call these methods “magic” because they are quite magical in how they work in PowerShell.  They don’t show up in Get-Member output, even if you apply -Force and request -MemberType All.  If you roll up your sleeves and dig in with reflection, you can find them; however, it requires a broad search because they are private extension methods implemented on a private class.  Yet even though they are not discoverable without peeking under the covers, they are there when you need them, they are faster than their older counterparts, and they include functionality that was not available in their older counterparts, hence the “magic” feeling they leave you with when you use them in PowerShell.  Unfortunately, these methods remain undocumented even today, almost a year since they were publicly released, so many people don’t realize the power that is available in these methods.  This article will try to correct that by explaining where they can be used and how they work so that you can leverage this magic when you use PowerShell.

A note about PowerShell 3.0

Before I get into explaining how the ForEach and Where methods work, I need to mention something with respect to these two methods and PowerShell 3.0.  While it is true that the ForEach and Where methods were only made available in PowerShell 4.0 and later versions, PowerShell 3.0 is still very widely used in many environments, and unless you’re using PowerShell in an environment that has standardized on PowerShell 4.0 and later, you may find yourself wishing you could take advantage of the syntax provided by the new methods when using PowerShell 3.0.  I felt this was a limitation worth addressing, so as part of the TypePx module that I recently published on GitHub and in the PowerShell Resource Gallery (aka the PowerShellGet public repository, currently in limited preview), I included ForEach and Where script methods that are functionally equivalent to the methods introduced in PowerShell 4.0 so that you can leverage the new syntax and functionality even if you’re using PowerShell 3.0.  There are a few shortcomings in this implementation, which I will highlight later in this article.

The ForEach method

ForEach is a method that allows you to rapidly iterate through a collection of objects and take some action on each object in that collection.  This method provides faster performance than its older counterparts (the foreach statement and the ForEach-Object cmdlet), and it also simplifies some of the most common actions that you may want to take on the objects in the collection.  Any objects that are output by this method are returned in a generic collection of type System.Collections.ObjectModel.Collection`1[psobject].

There are six supported ways that you can invoke this method, and each of these will be explained in greater detail below.  The supported arguments that can be used when invoking the ForEach method are as follows:

• ForEach(scriptblock expression)
• ForEach(type convertToType)
• ForEach(string propertyName)
• ForEach(string propertyName, object[] newValue)
• ForEach(string methodName)
• ForEach(string methodName, object[] arguments)
• ForEach(scriptblock expression, object[] arguments)

Note that these are supported argument pairings, not different overloads available for the ForEach method.  Using any argument pairings other than these may result in errors that do not clearly identify what the actual problem is.

ForEach(scriptblock expression) and ForEach(scriptblock expression, object[] arguments)

If you pass a script block expression into the ForEach method, you are able to perform the same kind of tasks that you would in a script block that you would use with the foreach statement or the ForEach-Object cmdlet.  Also, like the ForEach-Object cmdlet, the $_ and $PSItem variables both reference the current item that is being processed.  Any arguments that you provide beyond the initial script block argument will be used as arguments for the script block. This is just like how the -ArgumentList parameter works on the ForEach-Object cmdlet. Here is an example demonstrating how you might use this to execute a script block on each item in a collection:

# Get a set of services
$services = Get-Service c*
# Display the names and display names of all services in the collection
$services.foreach{"$($_.Name) ($($_.DisplayName))"}
# Select a property name to expand using a script block argument
$services.foreach({param([string]$PropertyName); $_.$PropertyName}, 'DisplayName')

You may have noticed something odd about this syntax, because I didn’t wrap the script block itself in brackets.  You can wrap it in round brackets, however that is optional in PowerShell 4.0 or later because the PowerShell parser was enhanced to allow for the brackets to be omitted whenever you are invoking a method that accepts a single script block argument.  Also, like the foreach statement and the ForEach-Object cmdlet, the script block that you provide is invoked in the current scope.  That means that any variable assignments you make inside of that script block will persist after the ForEach method has finished executing.

ForEach(type convertToType)

Unique to the ForEach method, you can pass a type into the ForEach method if you want to convert every item in a collection into another type.  For example, imagine you have a collection of objects and you want to convert those objects into their string equivalent.  Here is what that would look like with the ForEach method:

# Get a collection of processes
$processes = Get-Process
# Convert the objects in that collection into their string equivalent
$processes.foreach([string])

You could have performed the same task by typecasting the collection into an array of type string (e.g. [string[]]$processes), and typecasting the array is in fact significantly faster, however there’s a very good chance you wouldn’t even notice the difference in execution time unless you were working with a very, very large collection.  Despite the time difference, I will tend to prefer the ForEach method syntax in certain situations if it allows me to maintain elegance in the implementation by avoiding extra round brackets in the scripts I write.

ForEach(string propertyName)

In PowerShell 3.0 and later, a second parameter set was added to ForEach-Object to allow you to more easily retrieve the value of a specific property by simply passing in a property name as the only parameter value for ForEach-Object.  This convenience has been offered in the ForEach method as well.  Here is an example demonstrating how you can iterate through a collection and return a property of that collection:

# Get all services whose name starts with "w"
$services = Get-Service w*
# Return the names of those services
$services.foreach('Name')

Of course since version 3.0 of PowerShell, you could simply invoke _$services.Nam_e to get the names of all services, and that will complete faster than the ForEach method alternative (although you’ll only notice the performance difference in very large collections in the order of hundreds of thousands of objects); however, that only works for properties that are not on the collection itself, and it is a syntax that some scripters are not comfortable with due to the implicit nature of what the command does. The new ForEach method syntax provides you with a more explicit alternative that has the added benefit of being a little more self-documenting as well.

ForEach(string propertyName, object[] newValue)

Not only can you retrieve a property on a collection of objects, you can set a property on a collection of objects as well. This is functionality that is not available in the other foreach’s, unless you explicitly create the script block to do so. To set the property, you simply provide the property name and the value you want to use when setting that property, like this:

# Note, this is not a realistic example
# This would be used more commonly on configuration data
$services = Get-Service c*
# Now change the display names of every service to some new value
$services.foreach('DisplayName','Hello')

Just like assignments you would make using the equals operator, PowerShell will attempt to convert whatever you provide as the new value into the appropriate type for the property being assigned.

ForEach(string methodName) and ForEach(string methodName, object[] arguments)

To invoke a method, you simply provide the method name as the first argument, and then the arguments for that method as the second, third, fourth, etc. arguments. If the method does not take any arguments, you can simply pass in the method name and it will be invoked without any arguments. Here’s an example showing how you could kill a bunch of processes running a specific program:

# Get all processes running Chrome
$processes = Get-Process -Name Chrome
# Now kill all of those processes
$processes.foreach('Kill')

Here’s another example, this time using a method with arguments while showing how you could verify that commands are following best practices by using appropriate names and aliases for commonly used parameters:

# Get all commands that have a ComputerName parameter
$cmds = Get-Command -ParameterName ComputerName
# Now show a table making sure the parameter names and aliases are consistent
$cmds.foreach('ResolveParameter','ComputerName') | Format-Table Name,Aliases

As you can see from those results, there are definitely some inconsistencies in the implementation of ComputerName parameters that should be corrected.

That covers all of the functionality that is currently available in the ForEach method. As you can see, there is not a lot of new functionality offered in this method, but the syntax improvements when you are performing a simple task on a collection of objects are nice, and the ForEach method performance improvements when compared to the equivalent foreach statement for ForEach-Object pipeline are definitely a welcome improvement as well. With that explanation out of the way, let’s move on to the Where method.

The Where method

Where is a method that allows you to filter a collection of objects.  This is very much like the Where-Object cmdlet, but the Where method is also like Select-Object and Group-Object as well, includes several additional features that the Where-Object cmdlet does not natively support by itself. This method provides faster performance than Where-Object in a simple, elegant command.  Like the ForEach method, any objects that are output by this method are returned in a generic collection of type System.Collections.ObjectModel.Collection`1[psobject].

There is only one version of this method, which can be described as follows:

Where(scriptblock expression[, WhereOperatorSelectionMode mode[, int numberToReturn]])

As indicated by the square brackets, the expression script block is required and the mode enumeration and the numberToReturn integer argument are optional, so you can invoke this method using 1, 2, or 3 arguments.  If you want to use a particular argument, you must provide all arguments to the left of that argument (i.e. if you want to provide a value for numberToReturn, you must provide values for mode and expression as well).

Where(scriptblock expression)

The most basic invocation of the Where method simply takes a script block expression as an argument.  The script block expression will be evaluated once for each object in the collection that is being processed, and if it returns true, the object will be returned by the Where method.  This is the functional equivalent of calling the Where-Object cmdlet and passing it a script block.  Like the Where-Object cmdlet, the $_ and $PSItem variables can be used to refer to the current item that is being processed while inside of the script block.

Here is a very simple example, showing how you could get a list of running services.

# Get all services
$services = Get-Service
# Now filter out any services that are not running
$services.where{$_.Status -eq 'Running'}

This doesn’t offer any new functionality, but it offers much faster performance than Where-Object and the syntax is quite easy to follow, so you really should consider this for your scripts when you are performing client-side filtering of collections that you have stored in a variable.

Where(scriptblock expression, WhereOperatorSelectionMode mode[ ,int numberToReturn])

When you start looking at the optional parameters for the Where method, things start getting much more interesting.  Windows PowerShell version 4.0 included a new enumeration with a typename of System.Management.Automation.WhereOperatorSelectionMode.  Note the suffix of that typename: “SelectionMode”.  It is used to provide powerful selection capabilities in a Where syntax.  Here are the values included in this enumeration, along with their definitions:

Each of these offers some unique value when you are processing collections of data, so I’ll provide more details of each selection mode below.

Default

Not surprisingly, the default value of the mode argument is ‘Default’.  The default selection mode offers the same functionality that you get when you don’t provide a selection mode at all.  For example, we could have written the last line of our previous example like this:

# Now filter out any services that are not running
$services.where({$_.Status -eq 'Running'},'Default')

In that example, the extra argument isn’t necessary though, because it does the exact same thing that it would do if you didn’t provide the argument.  You could also provide the maximum number of objects that you want to return while using Default selection mode using the numberToReturn argument, like this:

# Get the first 10 services in our collection that are running
$services.where({$_.Status -eq 'Running'},'Default',10)

It is important to note that exact functionality is also available when using the First selection mode (which we’ll talk about in a moment), so it really isn’t practical to use any of the optional parameters at all when you are using the Default selection mode.

First

As you might have guessed, the First selection mode allows you to select the first object(s) in the collection that pass the script block expression filter. When you use First without a value for the numberToReturn argument, or when you use First with a value of 0 for the numberToReturn argument, only the first object that passes the filter will be returned.  You can optionally specify how many objects to return in the numberToReturn argument, in which case that many objects will be returned (assuming there are that many objects that pass the filter).

Here are some examples using our services collection showing the First selection mode in action:

# Get the first service in our collection that is running
$services.where({$_.Status -eq 'Running'},'First')
# Get the first service in our collection that is running
$services.where({$_.Status -eq 'Running'},'First',1)
# Get the first 10 services in our collection that are running
$services.where({$_.Status -eq 'Running'},'First',10)

Note that the second command in these examples returns the same results as the first command because it is simply explicitly passing in the default value of the numberToReturn argument when the First selection mode is used.

Last

The Last selection mode functions much like the First selection mode, allowing you to select the last object(s) in the collection that pass the script block expression filter.  When you use Last without a value for the numberToReturn argument, or when you use Last with a value of 0 for the numberToReturn argument, only the last object that passes the filter will be returned.  You can optionally specify how many objects to return in the numberToReturn argument, in which case that many objects will be returned (assuming there are that many objects that pass the filter).

Here are some examples using our services collection showing the Last selection mode in action:

# Get the last service in our collection that is running
$services.where({$_.Status -eq 'Running'},'Last')

# Get the last service in our collection that is running
$services.where({$_.Status -eq 'Running'},'Last',1)

# Get the last 10 services in our collection that are running
$services.where({$_.Status -eq 'Running'},'Last',10)

Also like the First selection mode examples, the second command in these examples returns the same results as the first command because it is simply explicitly passing in the default value of the numberToReturn argument when the Last selection mode is used.

SkipUntil

The SkipUntil selection mode allows you to skip all objects in a collection until you find one that passes the script block expression filter.  Once you find an object that passes the filter, SkipUntil mode will either return all objects remaining in the collection if no value or a value of 0 was provided to the numberToReturn argument, or it will return the first N remaining objects in the collection if a value greater than zero was provided to the numberToReturn argument.  In both cases, the results will include the first object that passed the filter.

Here are some examples using a subset of our services collection to show the SkipUntil selection mode in action:

# Get a collection of services whose name starts with "c"
$services = Get-Service c*
# Skip all services until we find one with a status of "Running"
$services.where({$_.Status -eq 'Running'},'SkipUntil')
# Skip all services until we find one with a status of "Running", then
# return the first 2
$services.where({$_.Status -eq 'Running'},'SkipUntil',2)

Until

The Until selection mode provides the opposite functionality of the SkipUntil selection mode.  It allows you to return objects in a collection until you find one that passes the script block expression filter.  Once you find an object that passes the filter, the Where method stops processing objects.  If you don’t provide a value for the numberToReturn argument, or if you provide a value of 0, the Until selection mode will return all objects in the collection leading up to the first one that passes the script block expression filter.  If you do provide a value for the numberToReturn argument that is greater than 0, the Until selection mode will return at most that number of objects, meaning that it may not even find an object that passes the script block expression filter.

Here are some examples using a different subset of our services collection to show the Until selection mode in action:

# Get a collection of services whose name starts with "p"
$services = Get-Service p*
# Return all services until we find one with a status of "Stopped"
$services.where({$_.Status -eq 'Stopped'},'Until')
# Return the first 2 services unless we find one with a status of
# "Stopped" first
$services.where({$_.Status -eq 'Stopped'},'Until',2)

Split

Split selection mode is unique.  Instead of returning a subset of the collection you start with in a new collection, it returns a new collection that internally contains two separate collections.  What those nested collections contain depends on how you use Split selection mode.  Split allows you to split a collection of objects into two.  By default, if you don’t provide a value for the numberToReturn argument or if you provide a value of 0 for the numberToReturn argument, Split will place all objects that pass the script block expression filter into the first nested collection, and all other objects (those that don’t pass the script block expression filter) into the second nested collection.  If you do provide a value greater than 0 for the numberToReturn argument, split will limit the size of the first collection to that maximum amount, and all remaining objects in the collection, even those that match the script block expression filter, will be placed into the second collection.

Here are some examples showing how Split selection mode can be used to split up a collection of objects different ways:

# Get all services
$services = Get-Service
# Split the services into two groups: Running and not Running
$running,$notRunning = $services.Where({$_.Status -eq 'Running'},'Split')
# Show the Running services
$running
# Show the services that are not Running
$notRunning
# Split the services into the same two groups, but limit the Running group
# to a maximum of 10 items
$10running,$others = $services.Where({$_.Status -eq 'Running'},'Split',10)
# Show the first 10 Running services
$10running
# Show all other services
$others

As you can see from this example, Split is quite a powerful selection mode, providing a mix of filtering, grouping, and selection in a single command call.

This collection of selection modes makes the Where method fast and elegant yet at the same time more powerful than the Where-Object, Group-Object and Select-Object combined in a single pipeline.  What’s not to love about that?

Shortcomings in the ForEach and Where script methods in TypePx

As I mentioned earlier in this article, I have written type extensions for PowerShell 3.0 and later and packaged them up into a module called TypePx.  TypePx is a script module that was written entirely in PowerShell, and it runs on PowerShell 3.0 or later.  If you are using PowerShell 3.0 (and only if you are using PowerShell 3.0), TypePx defines ForEach and Where script methods that mimic the behaviour of the ForEach and Where methods in PowerShell 4.0 and later.  While the extended type system in PowerShell makes it possible to mimic this behaviour, there are a few shortcomings due to the implementation being in PowerShell that impacted how far I was able to go with these type extensions.  This section will describe some of the differences and limitations that exist in the ForEach and Where script methods in TypePx that you may want to be aware of if you are using PowerShell 3.0.

Script blocks invoked from a script method run in a child scope

Unlike the ForEach and Where methods implemented as part of PowerShell 4.0 or later which invoke the expression script block in the current scope where the method is called, the ForEach and Where script methods implemented in PowerShell 3.0 invoke the expression script block in a child scope.  This is a limitation in PowerShell that has been there since the very beginning (a limitation, I might add, that I think is by far the biggest shortcoming of PowerShell as a language).

Due to this limitation, any variables that you assign inside of an expression script block will only be modified in the child scope.  This has implications if your expression script block is intended to update a variable in the scope in which you invoke ForEach or Where.  It is unlikely this would cause a problem while using Where because it is not very common to modify variables in a Where expression script block, but in ForEach script blocks this may pose a problem so you need to keep this in mind if you use these extensions.

I should note that I would like to remove this limitation altogether, and I believe I will be able to do so, however at the time I wrote this article I have not yet implemented a fix for this.

Most, but not all collections will have ForEach and Where methods

In PowerShell 4.0 and later, the ForEach and Where methods are magically made available for all types that implement IEnumerable except for String, XmlNode, and types that implement IDictionary.  In PowerShell, the extended type system does not allow extensions to be created for interfaces, only for types.  This is a challenge if you want to create a broad extension like ForEach and Where.  In the current implementation of these extensions in TypePx, the TypePx module finds all types in all assemblies loaded in the current app domain, and for all non-generic types that define IEnumerable but not IDictionary (excluding String and XmlNode), plus for all generic types that define IEnumerable but not IDictionary for generic collections of PSObject, Object, String, Int32, or Int64, the ForEach and Where script methods will be created.

This covers a large number of types that in my own testing has been sufficient, however you may run into types where you want to use these methods and they are not available.  If that is the case, let me know through GitHub and I’ll see what I can do.  This is also a limitation I would like to remove, but I need more time to research how to implement the equivalent functionality in a compiled assembly where I may be able to define it more like it is defined in PowerShell 4.0 and later.

PowerShell scripts are not as fast as compiled code

This probably goes without saying, but when you write something in PowerShell, which is an interpreted language, it will not run as quickly as it would if you wrote the equivalent logic in a language like C#.  This is absolutely the case for the ForEach and Where script methods, which internally use ForEach-Object, Where-Object, and pipelining to mimic the behaviour of the native ForEach and Where methods.  In this case, the advantage of having these commands comes from the elegant syntax and functionality they provide, plus being able to use these in scripts for PowerShell 3.0 and 4.0.  The performance benefits in ForEach and Where are only in the PowerShell 4.0 native implementation.

PowerShell 3.0 requires brackets around all method parameters

I mentioned in the examples above that I was able to invoke a method with a single literal script block parameter without wrapping that literal script block in additional brackets.  That capability exists only in PowerShell 4.0 or later due to improvements that were made to the parser in that release.  In PowerShell 3.0, the parser does not support this, so brackets are always required in order for the ForEach and Where script methods to work with a single literal script block parameter in that version.

Conclusion

When I started trying to mimic the behaviour of the ForEach and Where magic methods in PowerShell 3.0, I didn’t quite realize how much functionality they provided.  Digging into the technical details behind these methods so that I could create the extensions I wanted in TypePx helped uncover all of the hidden features in these very powerful methods, and I am very happy to share those with you in this article.  I hope this information helps you leverage this wonderful new set of features in your PowerShell work, even if you’re still using PowerShell 3.0.  Happy scripting!