The NuGet Team Blog

Where else is NuGet used in Visual Studio?

2013-05-20T00:00:00-07:00

All NuGet users are probably familiar with both the Manage NuGet Packages dialog and the Package Manager Console. These are the main user interfaces for installing and managing packages.

However, NuGet is also used in a couple of more subtle places. Below are a few of them that I am aware of. With the exception of the New Project dialog, these dialogs/commands invoke NuGet through the well-documented MEF services that NuGet exports.

New Project dialog

When you create a new Web Application project, either a Web Forms or MVC app, the project template invokes NuGet to install a set of pre-defined packages into the new project.

This feature is publicly documented here: http://docs.nuget.org/docs/reference/packages-in-visual-studio-templates. Which means it’s not only available to Microsoft templates, but to any Visual Studio template.
Add Service Reference dialog

When you open the Add Service Reference dialog to add a reference to a WCF Data Service endpoint, the tool uses NuGet to install the latest version of the Microsoft.Data.Services.Client package and its dependencies.
Publish dialog

If you attempt to publish your web app to Windows Azure, the Publish dialog detects if your project is using SqlMembership provider, and if so, will ask you to upgrade to Universal provider in order to work with SQL Azure. If you accept, it will install the Microsoft.AspNet.Providers and Microsoft.AspNet.Providers.Core packages, as well as configure your web.config to use the Universal provider.
Enable Azure Authentication command

If you enable Windows Azure Active Directory authentication in your web app, the command will install the System.IdentityModel.Tokens.ValidatingIssuerNameRegistry package.
Add Windows Azure Cloud Service project command

The Add Windows Azure Cloud Service project command, which appears on the context menu for Web projects, uses NuGet to install the WindowsAzure.Storage and Microsoft.WindowsAzure.ConfigurationManager packages.

I’m very happy to see various teams in Microsoft take advantage of NuGet to implement their own features. I expect even more widespread usage of NuGet in future versions of Visual Studio.

-Luan

Play with Packages, programmatically!

2013-05-20T00:00:00-07:00

One of the many often seen requests, from developers using NuGet, is examples on our NuGet core APIs. The requests range from finding meta-data for the packages to installing the package programmatically. In this blog post, I will provide a few examples of using NuGet Core APIs.

Getting Started

To use NuGet Core APIs, install NuGet.Core NuGet package in your project and add NuGet namespace to your using statements.

Get me the pre-released versions of a package

Let’s say you would like to get the list of all pre-released versions of EntityFramework package. Below is the code snippet that would get you the list using NuGet API v2.

//ID of the package to be looked up
string packageID = "EntityFramework";
		
//Connect to the official package repository
IPackageRepository repo = PackageRepositoryFactory.Default.CreateRepository("https://packages.nuget.org/api/v2");
        
//Get the list of all NuGet packages with ID 'EntityFramework'       
List<IPackage> packages = repo.FindPackagesById("packageID").ToList();

//Filter the list of packages that are not Release (Stable) versions
packages = packages.Where (item => (item.IsReleaseVersion() == false)).ToList();

//Iterate through the list and print the full name of the pre-release packages to console
foreach (IPackage p in packages)
{
	Console.WriteLine(p.GetFullName());
}

Help me get a package

Let’s say you would like to programmatically download and unzip a package in the path you want. Below is the code snippet that you would use to download and unzip the package in the path specified. The example below downloads and unzips EntityFramework 5.0.0 to one of my projects.

//ID of the package to be looked up
string packageID = "EntityFramework";
		
//Connect to the official package repository
IPackageRepository repo = PackageRepositoryFactory.Default.CreateRepository("https://packages.nuget.org/api/v2");

//Initialize the package manager
string path = <PATH_TO_WHERE_THE_PACKAGES_SHOULD_BE_INSTALLED>
PackageManager packageManager = new PackageManager(repo, path);
        
//Download and unzip the package
packageManager.InstallPackage(packageID, SemanticVersion.Parse("5.0.0"));

It’s quite easy, have fun!

As illustrated by the above examples, trying to work with NuGet packages programmatically is quite easy. Start exploring them today!

A quick note about the new MinClientVersion attribute

2013-04-30T00:00:00-07:00

In the NuGet 2.5 release, we added the MinClientVersion property to the package manifest schema to allow for a package to demand the minimum version of the NuGet client that can install it.

Because this feature was added in version 2.5, older versions of NuGet won’t be able to recognize it. Therefore, older clients will always refuse to install packages with the MinClientVersion property set, even if you set it to a value smaller than the version of the client.

For example, if you set MinClientVersion to 1.8, and the user tries to install your package with NuGet 2.2, the installation will fail even though 1.8 < 2.2. The user will see the following error message:

The schema version of ‘MyPackage’ is incompatible with version 2.2.1.0 of NuGet. Please upgrade NuGet to the latest version from http://go.microsoft.com/fwlink/?LinkId=213942.

In sum, this feature will grow in its utility over future NuGet iterations, starting with version 2.6

-Luan

Support for Native Projects

2013-04-26T00:00:00-07:00

In the NuGet 2.5 release notes, we called out that NuGet now has support for native projects. We wanted to explain exactly what “support for native projects” means.

Visual Studio Support

Until NuGet 2.5, if you were working in a Visual C++ project, the ‘Manage NuGet Packages’ context menu would not show up for the project; with NuGet 2.5, we now light this up. Any time we start supporting a new project type, we have to test to ensure all operations are working as expected. Each project system has slightly different behavior and this proved to be true with Visual C++ as well. We made the changes necessary to get NuGet install/update/uninstall operations working.

MSBuild Integration

C++ projects tend to have many different configurations–more than what NuGet is able to handle. To address NuGet’s configuration limitations, we are relying heavily on MSBuild properties and targets for native packages. These MSBuild properties and targets do the heavy lifting of providing references at build time, based on your project’s configuration.

To make MSBuild integration better, NuGet has created a new convention for automatically importing MSBuild properties and targets from a NuGet package. Alongside the existing \content, \lib, and \tools folders, NuGet now recognizes a new top-level folder: \build.

Within the \build folder, you can provide a “.props” file and/or a “.targets” file that will be automatically imported into the project. For this convention, the file name must match your package id with either a “.props” or “.targets” extension. For example, the ’cpprestsdk’ package includes a ‘cpprestsdk.targets’ file in its \build folder. Files with the “.props” extension will be imported at the top of the project file, and files with the “.targets” extension will be imported at the bottom of the project file.

Note that this \build folder can be used for all NuGet packages and not just native packages. The \build folder respects target frameworks just like the \content, \lib, and \tools folders do. This means you can create a \build\net40 folder and a \build\net45 folder and NuGet will import the appropriate props and targets files into the project. You no longer need to write PowerShell install.ps1/uninstall.ps1 scripts to import MSBuild targets files!

Native Target Framework

When targeting native projects, a new target framework name is now recognized: native. There is no versioning within the ‘native’ target framework; NuGet treats all C++ projects the same.

We expect that most native packages will be produced using the \build\native folder, with at least a targets file and then a props file if needed. The ‘native’ target framework is also recognized for the \content and \tools folders, but because NuGet cannot directly add references to native projects, the ‘native’ target framework is not recognized within the \lib folder.

Creating Native Packages

If you’re interested in publishing your native libraries via NuGet, you can choose to create the NuGet packages manually. However, there’s an easier way–the CoApp project volunteered to write C++ oriented tools to create NuGet packages, and they have released a beta version of their tools. These tools simplify the process of generating the MSBuild files and NuGet packages from existing header and library files–you just need to create a configuration script to describe the contents of the package and then run the tools to generate the NuGet package.

There are tutorials, reference documentation, and a video on how to use the tools to create NuGet packages. The CoApp project organizers also plan to create and publish packages for several open source libraries into the NuGet Gallery over the next several weeks.

We’re encouraging people to publish packages with the “native” tag to make it easier to find packages you can use in C++ projects. You can search the NuGet Gallery using “tag:native” to find packages tagged with “native”. This works from both nuget.org and within Visual Studio. Note that not all packages tagged with “native” are actually C++ projects though.

The CoApp and Openness@Microsoft blogs have additional information about this project.

Feedback

Please leave us feedback here on the blog–we’d love to hear your experiences trying out NuGet in your C++ projects, and whether there are additional features we could add to make this work even better for you. This is just the start of C++ integration in NuGet and we need your input on where we should take this in the future to make working with C++ libraries in Visual Studio easier.

NuGet 2.5 Released

2013-04-25T00:00:00-07:00

Download it now

NuGet was included in all editions of Visual Studio 2012 and it’s also available for Visual Studio 2010 through the Visual Studio Extension Manager. To get the latest version of NuGet installed in Visual Studio, either check for updates in the extension manager or download the latest version directly.

Acknowledgements

We would like to thank the following external contributors for their significant contributions to NuGet 2.5:

Daniel Plaisted (dsplaisted) - Add MonoAndroid, MonoTouch, and MonoMac to the list of known target framework identifiers. #2847
Andres G. Aragoneses (knocte) - Fix spelling of NuGet.targets for a case-sensitive OS #2865
David Fowler (dfowler) - Make the solution build on Mono.
Andrew Theken (atheken) - Fix unit tests failing on Mono.
Olivier Dagenais (OliIsCool) - nuget.exe pack command does not propagate Properties to msbuild #2920
Miroslav Bajtos (MiroslavBajtos) - Modified XML handling code to preserve formatting. #1511
Adam Ralph (adamralph) - Added recognized words to custom dictionary to allow build.cmd to succeed.
Bruno Roggeri (broggeri) - Fix unit tests when running in localized VS.
Gareth Evans (garethevans) - Extracted interface from PackageService
Maxime Brugidou (brugidou) - Handle project dependencies when packing #936
Xavier Decoster (XavierDecoster) - Support Clear Text Password when storing package source credentials in nuget.cofig files #2991, #3164

We also appreciate the following individuals for finding bugs with NuGet 2.5 Beta/RC that were approved and fixed before the final release:

Tony Wall (CodeChief) - MSTest broken with lastest NuGet 2.4 and 2.5 builds #3200

What’s Included in NuGet 2.5

This release was so big, we felt compelled to skip versions 2.3 and 2.4! To date, this is the largest release we’ve had for NuGet, with 169 work items in the release.

For details about what’s in the release, the release notes are published on docs.nuget.org.

Here are the most notable features:

Allow users to overwrite content files that already exist
Automatic import of msbuild targets and props files
Specify different references per platform using element
Update All button to allow updating all packages at once
Improved project reference support for NuGet.exe Pack
Add a ‘Minimum NuGet Version’ property to packages
Dependencies are no longer unnecessarily updated during package installation
NuGet.exe outputs http requests with detailed verbosity
NuGet.exe push now supports UNC and directory sources
NuGet.exe supports explicitly-specified Config files
Support for Native projects

NuGet’s description

We’ve outgrown NuGet’s current description. According to nuget.org, we have the following description and slogan:

Jump Start Your Projects with NuGet

NuGet is a Visual Studio extension that makes it easy to install and update third-party libraries and tools in Visual Studio.

So install NuGet and get a jump on your next project!

We have come up with a new description for NuGet:

NuGet is the package manager for the Microsoft development platform including .NET. The NuGet client tools provide the ability to produce and consume packages. The NuGet Gallery (nuget.org) is the central package repository used by all package authors and consumers.

You can expect the home page at nuget.org to reflect this new description soon.

XDT web.config transforms

2013-04-22T00:00:00-07:00

Sitting there since October 12, 2010, with a total of 55 votes accumulated since then, there’s a NuGet feature that has had more than its fair share of attention: Support Visual Studio (XDT) Web.config Transforms. It’s the 17th highest voted work item of all time for NuGet, and the 5th highest voted open work item. Today, we are announcing that we are finally unblocked on getting this feature integrated into NuGet!

Last November, we made the decision to act on this feature request, and Luan Nguyen (dotnetjunky) posted a discussion thread that NuGet would be taking a dependency on the XDT assembly. Within a few hours, Andreas Håkansson (thecodejunkie) pointed out a really good glitch in our plan: we needed to make sure the feature would work on Mono!

Fast forward 4 months, and we have resolution for the requirement of XDT support working on Mono–Microsoft has open-sourced the XML Document Transform (XDT) library at http://xdt.codeplex.com, and NuGet will be taking a source dependency on XDT rather than a binary dependency.

Here’s the net result of the work we’ve done with the team that develops the XDT library:

A Microsoft.Web.Xdt NuGet package is published to nuget.org.
The NuGet team used that NuGet package to prototype integration of NuGet and XDT.
The XDT library is open-sourced (not accepting contributions) at xdt.codeplex.com.
The NuGet team will now use the XDT source to provide the integration.
This allows all NuGet users on all platforms to use the feature. NuGet can be built on Mono and have the XDT source built there too.

NuGet package authors will be able to create two new files in their packages to perform XDT transforms:

web.config.install.xdt (and yes, files other than web.config are supported)
web.config.uninstall.xdt

These files can use XDT functionality to transform XML files during install, and then to reverse the transformations upon uninstall. It’s a concept very similar to install.ps1/uninstall.ps1, but using the XDT library for the XML transforms.

We are working on the source-based integration of this feature now, and we plan to include it in the NuGet 2.6 release in early summer 2013.

For more information about the XDT library and its open source project, check out the .NET Web Development and Tools blog post.

Goodbye preview.nuget.org. Hello staging.nuget.org!

2013-04-19T00:00:00-07:00

TL;DR The preview.nuget.org site is going away. Switch to staging.nuget.org by April 26th!

Goodbye preview.nuget.org

We have a version of the NuGet Gallery called “Preview” and it’s been hosted at preview.nuget.org since early in the Gallery’s lifetime. Until recently, our QA team used preview.nuget.org to do their testing, but we’ve also occasionally advised package owners to use the site as a sandbox environment. Over time, this has put us in the situation of the “preview” site serving two different purposes: 1) it was an internal QA testing environment, and 2) it was a customer sandbox/preview environment.

These two use cases for the environment have drastically different requirements. The QA environment needs to be sanitized to not have any real data and allow the QA team to do whatever they need to the data at any time. The customer sandbox environment needs to have a mirror of recent production data to provide a useful staging environment. Because of these conflicting requirements, we are separating our QA testing environment from the sandbox, and we’ll be decommisioning preview.nuget.org on April 26th.

Hello staging.nuget.org

To provide a sandbox where package authors can stage their packages to test their publishing, we are creating a new staging environment at staging.nuget.org. Staging is a very close mirror of production and its data is frequently copied from production (although not yet on any fixed schedule).

If you have packages you want to publish and you’re nervous about the publishing process, please use staging.nuget.org. Your account information and API key are not guaranteed to be in sync because the data is not a live mirror, but you can register and log on to Staging just like you would on http://nuget.org. You can then push your packages to Staging and ensure you have your publishing process and your package metadata just right before publushing to nuget.org.

NuGet 2.5 Release Candidate

2013-04-15T00:00:00-07:00

Updated: The NuGet 2.5 Release Candidate was available from April 15 to April 25, 2013. For the several hundred of you that downloaded the release candidate, we truly appreciate your verification of the RC build that became the final release.

What’s Included in NuGet 2.5

This release was so big, we felt compelled to skip versions 2.3 and 2.4! To date, this is the largest release we’ve had for NuGet, with over 160 work items in the release.

For details about what’s in the release, the tentative release notes have been published to docs.nuget.org.

Here are the most notable features:

Allow users to overwrite content files that already exist
Automatic import of msbuild targets and props files
Specify different references per platform using element
Update All button to allow updating all packages at once
Improved project reference support for NuGet.exe Pack
Add a ‘Minimum NuGet Version’ property to packages
Dependencies are no longer unnecessarily updated during package installation
NuGet.exe outputs http requests with detailed verbosity
NuGet.exe push now supports UNC and directory sources
NuGet.exe supports explicitly-specified Config files
Support for Native projects

NuGet’s description and slogan

We think we’ve outgrown NuGet’s current description. According to nuget.org, we have the following description and slogan:

Jump Start Your Projects with NuGet

NuGet is a Visual Studio extension that makes it easy to install and update third-party libraries and tools in Visual Studio.

So install NuGet and get a jump on your next project!

We have come up with a new description for NuGet:

NuGet is the open-source package manager for the Microsoft platform and .NET/Mono. The NuGet client tools provide the ability to produce and consume packages. The NuGet Gallery (nuget.org) is the central package repository used by all package authors and consumers.

We’d like your feedback on this new description. And for a new slogan to replace “Jump Start Your Projects with NuGet,” we’re still undecided. We want something fun and maybe a little edgy. If you’d like to submit your idea, tweet your idea using the #nugetslogan hashtag. Once we have a good list to choose from, we’ll open up a poll. Follow @nuget for updates.

Tweets about "#nugetslogan"

Recent NuGet Gallery Updates

2013-04-11T00:00:00-07:00

The NuGet Gallery crew has gotten into a rhythm of publishing updates every 2 weeks. With the increased update frequency, we found that we needed to create a changelog to highlight new features. That changelog is now available directly from within our source repository on GitHub.

You can access the changelog directly here: https://github.com/NuGet/NuGetGallery/blob/master/Changelog.md

Here is the current log, showing the changes that have been made over the last 2 deployments.

April 4, 2013

Top 500 packages exposed in the feed

The nuget.org API (V2) feed now exposes the top downloaded packages (over the last 6 weeks). This can accessed be via url nuget.org/api/v2/stats/downloads. At this time, the top 500 packages are shown by default and that is also the maximum number returned.

You can limit the numbers of results using ?count in the query string. For example, nuget.org/api/v2/stats/downloads?count=10 would return the top 10 downloaded packages in last 6 weeks - with information like download count, gallery url and feed url for that package.

The default and maximum count of 500 might change over time, so we recommend always specifying a count parameter if you are programmatically consuming this data.

Numeric rank for packages stats

The Statistics page now shows the numeric rank of the package (based on the download count).

Links to gravatar in profile page

The profile editing page now includes help text and a link to gravatar making it easy for users to update their profile picture.

UserName optimization in DB (backend)

The “Users” table is optimized to have “UserName” as index for performance enhancements.

Other minor bug fixes

Complete list can be found here: Production Deployment 4/12

March 28, 2013

Support for MinClientVersion

You can now upload packages with ”minclientVersion” to the NuGetGallery.

The minclientVersion of the package will shown in the package home page right next to the package description.

Contacting support

The “Report Abuse” page has been revamped to enable users to chose the specific issue with the package they are reporting. It also guides the user to differentiate between “Contact Owners” and “Report Abuse”.

Improved package statistics

The package statistics now shows the break down of downloads based on the NuGet client (like NuGet CommandLine 2.1, NuGet Package Manager console 2.2 and so on. It also shows the split of the type of download operation (like Install, Restore, Update).

Other minor bug fixes

Complete list can be found here: Production Deployment 3/28

The March 6th NuGet Gallery Outage

2013-04-04T00:00:00-07:00

On March 6th 2013, nuget.org’s package download was broken for one and a half hours. Some outages involve factors beyond our control; this time it was very much our own fault and we would like to apologize. We also want to openly discuss the outage and reassure you that we are working on preventing reoccurences.

Timeline of events

The outage was triggered by the NuGet Gallery crew performing a regular planned upgrade to nuget.org. We had successfully completed the previous production upgrade as recently as 2 weeks prior (in fact we are now shipping code to nuget.org on regular 2-3 weeks intervals). The timeline of the outage was something like this:

2:00 - We assembled to do deployment to production. Following our existing process, we backed up the production database, reviewed the state of our master branch in git, and checked that our preproduction environment still looked healthy.

2:25 - We published the new code to the Production Web Role in Azure staging slots. Because we are using Entity Framework migrations at App Startup time, and because the VMs in staging slots run against the same database as the (still live) VMs in production slots, we did not feel safe testing these new role instances prior to doing the VIP swap which brings the new code live.

2:30 - We do the VIP swap, and the new code is now running nuget.org. We start doing sanity tests of production, and checking the error logs in case we will see evidence of any problems there.

A few minutes later, we started receiving tweets and emails stating that package downloads were failing. We acknowledged the issue on twitter. Related tweet. We really appreciate your tweets at @nuget when you encounter site issues!

2:35 - We realized gallery is redirecting package download requests to package URLs that don’t exist. We also realized why. The problem is in our production environment package blobs have not been updated to the new naming convention that the new code we just deployed requires.

We started running the batch job, which already exists, and has been used on preproduction environments. This will update the production environment package blobs and fix package downloads so the world can get their nupkgs. We estimated that this batch job will take 40-60 minutes to run.

While we monitored the progress of the batch job, we were simultaneously second guessing ourselves and trying to think of a safe way to get the site back up faster. Unfortunately VIP swapping back to the old production VMs is not a viable solution, the reason again being the way we are using Entity Framework migrations at App_Start which means our old production code is probably not going to be happy with a database which is in a state it does not understand.

2:40 - We got worried. The batch job seemed to be working very slowly, or not at all.

3:05 - We realized the batch job itself is broken. We fixed the code, and restarted the whole batch job. Again we monitored progress of the batch job. This time, eventually, we could verify it was working properly. We thought the batch job would take about an hour to complete, but it ended up taking 85 minutes. Packages gradually became available for download during this time, so some people may have seen their problems resolved sooner. Related tweet.

4:30 - We verified that the batch job had completed and all packages are back up and working. Related tweet.

The Problematic Code Changes

Here’s what motivated the change causing all these problems:

Package IDs in URLs on the nuget gallery are case insensitive.
Azure Blob Storage, which we use for holding all our nupkg files for download, is case sensitive.
Package IDs have their ‘canonical’ casing preserved in the NuGet database, and is used to build the filename that retrieves the blob from blob storage.
The canonical casing is determined the first time the package is published and it was immutable. Many package authors have wished to change the casing, but we couldn’t support it. This meant that any given version of a package might claim a package ID that is different from the canonical ID.

The consequence of 3. and 4. is that in order to service a package restore download request, which contains the package ID in casing which may be different from blob storage, nuget.org must query the database to retrieve the canonical ID before we could service the download request. As a further consequence package download stops working any time our database goes down.

Avoiding the database for package downloads

We updated the code that handles package download requests to look for packageID.ToLowerCase(); This means we no longer have to hit the database to retrieve package ID’s canonically cased version, we can just use ToLowerCase() on the package ID sent in the download request.

We apologize for the irony! We intended package downloads to become more robust but actually ended up breaking package downloads.

The Postmortem - What Went Wrong

We held a meeting the day after the outage, where we did a break-down starting with a summary of everything that went wrong, during the upgrade and outage. Here’s what went wrong.

The outage happened. We published a problem code to the VMs without the required associated data because we are missing a step in the deployment.
We tested the production environment after we published the code. This is when we discovered the issue. The time spent testing and discovering the issue contributes to the total delay between the problem occurring and us fixing the problem.
We were not confident we could safely revert the VIP swap to unpublish from production. If we could have done it then package downloads would have started working immediately, and it would have been probably a 5 minute downtime, not a 1.5 hour downtime. So our inability to safely revert is a problem.
The batch job for fixing the problem failed the first time. It contributed to the outage duration significantly the time to realize the script was broken and fix it again was 20 minutes or more.
And finally, the missed step of running the fixed batch job to recover the gallery just takes a long time because it’s doing a lot of work.

In a deeper analysis of how the problem code went into production in the first place without knowing that it would cause a problem, we found out that

The script had been run against our QA environment (called Preview) by the developer, without involving QA in testing the script.
The script had not been run against the preproduction environment (called Staging) ever. One reason for that is that Preview and Staging share the same Azure blob storage account, so it would have made no difference to observable behavior during testing.
Even though the need for running the script had been communicated verbally to the crew, because it was weeks earlier and because the step was not part of the deployment process for Preview and Staging, the step was completely forgotten about by the time it came to do the actual production deployment.
QA team had noticed issues downloading from Preview and Staging environments but not realize they were important because it was normal for download to be broken in these environments. The reason is these environments don’t have the full set of the production blob data in their own storage account.

Conclusions - What We Plan to Change

Plans for preventing and mitigating future nuget.org downtime caused by the NuGet gallery crew’s upgrades to nuget.org.

Ensure our deployment checklist of ALL steps required for deployment is updated alongside any code changes that require special deployment steps.
Validate production deployments using automated tests immediately after going live. We want to be able to detect the problems faster than it can be tweeted to us.
Expect problems during upgrades, and have a clear recovery plan in place for every deployment. The plan details how to quickly and safely recover when the deployment fails.

Ideally our recovery plan should always be VIP swap. We are currently experimenting with changing our use of Entity Framework to a model which will not depend on running migrations during the deployment itself.

A second recovery plan is database restore plus VIP swap. We would first set the Gallery to Read-Only mode (this is new) during the deployment, which means you will be unable to create accounts and upload packages while the deployment occurs.

Test the recovery plan in preproduction before doing the production deployment. (Confidence in a plan that has been dreamed up under pressure and not been tested is low!)
We need to make the QA and Preproduction environments squeaky clean and as reliable as production. We should never have issues that occur there that get ignored and thereby slip into production.
Stopping Preview and Staging from sharing the blob storage account will help us ensure we have done clean practice deployments.
Enhance our service monitoring to give us better feedback on when and how package download gets broken.

Improved search syntax on nuget.org and in NuGet Visual Studio extension

2013-03-25T00:00:00-07:00

The NuGet team recently deployed a new and very useful search capability on nuget.org, which allows you to narrow your search to a pariticular attribute of a NuGet package. For example, if you want to search for packages which contain ‘elmah’ within the Id, you type ’id:elmah’. (Note that this will execute a ‘contains’ search, not an exact match search. Which means it will return elmah, elmah.corelibrary, etc.)

Similarly you can use the same syntax to search for other attributes in the package metadata. Here’s the complete list of supported attributes:

Attribute	Example
id	id:jQuery
title	title:Validation
description	description:dependency injection
authors	authors:Outercurve Foundation
owners	owners:dotnetjunky
tags	tags:silverlight

Note that when a search query contains spaces, the gallery will perform an OR search against words in the query. AND search is currently not supported.

Even better, this feature is automatically available for any NuGet client that calls the Search service method from nuget.org. This can be seen in the Visual Studio Package Manager dialog as shown below.

Happy searching!

Introducing Package Statistics

2013-02-26T00:00:00-08:00

We have introduced a new feature on http://nuget.org where you can see download statistics for packages. With this initial drop, we hope to whet your appetite and collect your input on further statistics features; it’s a mere fraction of what we can imagine.

Statistics Tab

At the top of the site, a new ‘Statistics’ tab is displayed.

This tab takes you to http://nuget.org/stats, which shows the two high-level statistics reports we have at the moment. While we expect this page will expand with more reports, the first two we’ve introduced are “Downloaded Packages (Last 6 Weeks)” and “Downloaded Package Versions (Last 6 Weeks).”

Downloaded Packages

This report shows the total download numbers for all versions of a package. The full report includes the top 100 packages over the last 6 weeks. It’s no shock that packages like jQuery, Newtonsoft.Json, jQuery.Validation, and Microsoft.AspNet.Razor are at the top. But since the report is based on the last 6 weeks, instead of all time, we expect to see more changes on this report than on the full list of packages.

Downloaded Package Versions

This report shows the download numbers for a single version of a package. This full report includes the top 500 package versions over the last 6 weeks, and a package can show up multiple times if it has had more than one frequently downloaded version. From here, we can learn more about which versions of packages developers are using and how long it takes a new version to overcome an older one.

Version Breakdown for a Package

On the ‘Downloaded Packages’ report, the download count is a hyperlink. Following that hyperlink for a package will show you a report for the package’s downloads by version. This package-specific report lets you see the recent (6 weeks) downloads for any package, to further identify download trends for a package’s versions.

Statistics for any Package

When viewing the detail page for any package, there’s a new ‘Package Statistics’ link on the left side. This link takes you directly to the version breakdown for that package, showing the download statistics for the last 6 weeks. This data will be different from the download counts displayed directly on the package page, where that is a total of downloads for all time, instead of just the last 6 weeks.

Note that there are not any reports visible only to the package owner. Everyone can see the package statistics for any package.

Availability

As we first launch the package statistics feature, many packages will not have statistics available right away. You may see a message that states, “Download statistics are not currently available for this package, please check back later.” After the deployment of the feature, we are firing off the job to produce statistics reports for all packages. We expect this to take about a day. So if your package’s statistics aren’t yet available, just check back again later.

After the initial release of the feature, there will be a job running nightly to refresh every package’s statistics reports.

Feedback and Input

As mentioned, this is just the beginning of package statistics. We expect other reports to highlight which NuGet operation caused the download, whether it was a fresh Install, an Update, or a Package Restore. We are now collecting that data and in a handful of weeks we’ll be able to report on it. We also expect reports that bring in dependency graph data, packages that have been recently updated, and more.

If you have feedback or input for what statistics would be useful to you, either as a package consumer or package author, please let us know. The best way is to file an issue on the repository.

Workflow Changes for CodePlex Issues

2013-02-11T00:00:00-08:00

A couple weeks ago, some of you noticed that the core team closed a few issues on http://nuget.codeplex.com that were in the “Soonish” release. The items that were closed were feature suggestions that, while not bad ideas, were things that the core team decided weren’t going to make the cut for the next few releases. The pushback from the community in response was understandable. After all, if NuGet is an OSS project, why would the core team close good ideas simply because we weren’t going to act on them? Firstly, we’ve reopened the items that we had closed and moved them to the “Product Backlog” release (more on that momentarily). We also spent some time thinking about how we can do a better job balancing what feels like two competing interests in the same issue repository. The first is the interest of the core team to more efficiently manage its day to day work – and this means maintaining a high signal to noise ratio in the release buckets that we work from. The second interest is that of the community to actively participate in development of NuGet, whether that means filing new product ideas or pulling a new idea from the list which will ultimately find its way back into the product via a pull request. So here’s how we’re planning to organize issues and perform triage going forward in an attempt to better satisfy both sets of interests:

Get rid of the “Soonish” release. We had originally created this as a way to identify things that were potential candidates for a near term release as opposed to items in “Product Backlog” – however, over the last several months, we’ve observed that items in this release have built up to a level where the core team isn’t going to be able to act on many of these either – resulting in effectively 2 different backlog buckets. In sum, it was a nice idea, but it hasn’t worked out in the way that we had hoped.
Rename “Product Backlog” to “Up For Grabs”. This release represents issues that we would accept as a pull request if someone from the community implemented it with sufficient quality. One of the reasons that we closed items in “Soonish” rather than moving them to backlog was that we believe “backlog” to imply that the core team will eventually get around to acting on the item, and we felt like this was misleading. Items in backlog are those that the core team has decided are good ideas but not ideas that we will likely revisit. We do periodically pull highly-voted items from the backlog into an active release, but this is not a part of our regular rhythm. As such, we want the name of the release to more accurately reflect how it’s being used.
Add an acknowledgements.txt file to the repository and insert this text into the release notes. In general, we want to figure out better ways to thank individuals who take their role in the NuGet community to the next step by submitting pull requests for the issues that they are passionate about. Adding this kind of recognition seems like an obvious step that we can take here to do better at encouraging and rewarding this kind of participation.
We triage issues a couple times per week. When considering a new issue, the decision flow will look like the following:
- Is this something that we need to work on for the current release? If so, assign to the current release. The bar for this question changes based on the type of release (e.g. some releases are more focused on risk management, some on new feature development).
- Is this something that the core team plans to work on? If so, assign to the next actual release. No more of the revolving “Next” release as this would allow us to be passive about making decisions on issues that we repeatedly never get to.
- Is this something that we think is a good idea, but know that the core team is not going to get around to working on in the foreseeable future? If so, assign to “Up For Grabs” release so that it can be voted on and ideally picked up by the community.
- Is this something that we do not think is a good idea for NuGet? If so, close the issue with relevant explanation.
The planning process for a new release will look like the following:
- Ensure that roadmap features have been added to the upcoming release (e.g. ensure that all potential work for the release has been added in the form of a CodePlex issue).
- Triage all of the unfinished items in the release that was just completed. For each item, either move it to the next release, move to “Up For Grabs”, or close.
- If an issue has cascaded from release to release more than 2 times, move it to “Up For Grabs”

We’re sorry if the recent closing of issues on CodePlex gave you the impression that we were somehow making a move towards being less open and hope that the above steps give you a little more insight into how we think about our working and planning rhythms. The core team would love to be able to implement all of the good ideas that fit within the vision for NuGet - however, we are finite and can only get through a subset of the total items. With your help in grabbing issues from Up For Grabs, we’re looking forward to being able to get a lot more accomplished!

The quality of issues filed on NuGet has historically been very high, and we appreciate that. In the same way, we want and appreciate your thoughts and comments on how we can continually improve our processes - hopefully that is evident in these changes. As such, we’ll hold off for a couple of days to collect your feedback on the new approach and then, assuming that there are no blocking issues raised, make the changes outlined above.

Managing Package Owners on nuget.org

2013-02-05T00:00:00-08:00

While each NuGet package’s nuspec metadata defines the package’s owner(s), the NuGet gallery at nuget.org ignores that piece of nuspec metadata. Instead, ownership of a package is defined by who publishes the package to the gallery. Sometimes this metadata needs to be managed beyond the initial package publishing too, which means the owner metadata needs to be mutable while the package itself is not.

Setting a Package’s Initial Owner

When a brand new package is published to nuget.org, its initial owner is defined by the user that published the package.

If the website’s ‘Upload Package’ page was used to publish the package, then the logged-on user is the package’s owner.
If nuget.exe push was used to publish the package from the command-line, then the API key specified to nuget.exe will define the package’s owner. This can be done by either including the API key in the Push command, or by using the SetApiKey command.

Specifying Additional Package Owners

It’s common for a package to have multiple developers; each developer can be listed as a co-owner of the package. Packages can also utilize a convention for “organizations” to indicate company/group ownership of a package. For example, the Microsoft ASP.NET packages are co-owned by microsoft, aspnet, and sometimes individuals on the feature team. The ‘microsoft’ and ‘aspnet’ accounts are simply set up with a mailing list email address that reaches the teams that manage the accounts.

Once a package has been published to nuget.org, it’s easy to invite additional users to become owners of the package.

Log on to nuget.org with the account that is the current owner of the package;
Navigate to the package page using the ‘Packages’ tab, searching, or clicking your username to and then ’Manage My Packages’;
When logged on as the package’s owner, there is a ‘Manage Owners’ link on the left side to click;
Enter the username of the person to add as an owner and click ‘Add’;
An email is then sent to the new co-owner, as an invitation to become an owner of the package;
Once that user clicks the link, they are a full co-owner with full control over the package, including the ability to remove other users as owners.

Until the new owner confirms ownership, they will not be listed as an owner of the package. When viewing the ‘Manage Owners’ page, you will see a “pending approval” entry in the current owners. That invitation can be removed, just as other owners can be removed.

This process of invitations prevents users from falsely adding other users as owners of their packages. Note that the “Authors” metadata is purely freeform text; only “Owners” are controlled.

Removing Package Owners

All package owners are peers. This means any package owner can publish a new version of the package. It also means that any package owner can remove any other package owner. No owner has more authority than other owners.

When a package has multiple owners and one needs to be removed the process is simple:

Log on to nuget.org with the account that is the current owner of the package;
Navigate to the package page using the Packages tab, searching, or clicking your username to and then ’Manage My Packages’;
When logged on as the package’s owner, there is a ‘Manage Owners’ link on the left side to click;
Click the ‘remove’ link next to the owner to be removed.

Transfering Package Ownership

We sometimes get support requests to transfer package ownership from one user to another, but you can almost always accomplish this yourself. Transfering ownership from one user to another is simply a combination of the two features above.

The current owner invites the new user to become a co-owner and the new user accepts the invite;
The new user removes the old user from the list of owners.

This request has come in under a couple forms but the process works the same.

The package ownership is changing from one developer to another
The package was accidentally published using the wrong account

Orphaned Packages

One last scenario has occurred, but not many times. Packages have become orphans and the only package owner account cannot be used to add new owners. Here are some examples of this scenario:

The owner’s account is associated with an email address that no longer exists and the user has forgotten their password
The registered owner has left the company that produces the package and cannot be reached to update the package ownership
Due to a bug that has only affected a handful of packages, the package is somehow ownerless on the gallery

Some members of the NuGet team are in the ‘admin’ role on the gallery and can access the ‘Manage Owners’ link for any package. If you are the rightful owner of a package and cannot reach the current owner to gain ownership permissions, then use the ‘Report Abuse’ link on the gallery to reach the NuGet team. We will then follow a process to verify your ownership of the package. If we determine you should be an owner of the package, we will use the ‘Manage Owners’ link for the package ourselves and send you the invite to become an owner. We will only do this after verifying that you should be an owner and the process for this varies by circumstances. Often times, we will use the package’s Project URL to find a way to contact the project owner, but we may also use Twitter, Email, or other means for contacting the project owner.

A quick tutorial on the Update-Package command

2012-12-31T00:00:00-08:00

Among the supported commands in NuGet’s Package Manager Console, Update-Package is arguably the most powerful in terms of its supported parameters and switches. The primary purpose of the command is to update package(s) in your projects to a higher (possibly latest) version. However, the provided parameters allow you to twist this behavior in many different ways. In this post, I’m going to give a quick tutorial on what possible ways you can invoke the command.

In total, the Update-Package command offers eight parameters (in addition to the built-in PowerShell parameters):

Parameter	Type	Description
-Id	string	Specifies the Id of the package to be updated.
-ProjectName	string	Specifies the name of the project in which packages should be updated.
-Version	SemanticVersion*	Specifies the new target version of the package as a result of the update.
-Source	string	Specifies where to look for package updates, overriding the package sources that are specified in the Options dialog. This value can be either a url to a remote repository or a path to a local repository or the name of a package source specified in the Options dialog.
-IgnoreDependencies	Switch parameter	If set, NuGet will ignore dependency packages and only update the main package.
-Safe	Switch parameter	If set, NuGet will only update to a new version that has the same major and minor versions as the previous package. For example, if the old version is 1.2.0, NuGet will accept the update package with version of 1.2.1 or 1.2.9999 but it will not accept 1.3.0.
-IncludePrerelease	Switch parameter	If set, NuGet will consider prerelease packages as candidates for updates.
-Reinstall	Switch parameter	If set, instead of updating the package to the latest version, NuGet will uninstall the package and reinstall the same version. This is useful when, for example, you've updated the target framework of your project, e.g. from .NET 4.0 to .NET 4.5, and you want to reference .NET 4.5-specific assemblies in the package. You can't set this parameter together with the -Version parameter.

*Although the type of the -Version parameter is SemanticVersion, which is a type declared by NuGet, you can just use string when you set it. PowerShell will automatically convert it for you, as long as the value can be parsed by the SemanticVersion class.

With that, here are the common use cases for the Update-Package command:

Update a particular package in a project to the latest version:
Update-Package jQuery -ProjectName MyProject
Update a particular package in a project to the latest version, using safe update rule:
Update-Package jQuery -ProjectName MyProject -Safe
Update a particular package in a project to a particular version:
Update-Package jQuery -ProjectName MyProject -Version 1.8
Update a particular package in all projects of the current solution to the latest version:
Update-Package jQuery
Update a particular package in all projects of the current solution to a particular version:
Update-Package jQuery -version 1.8
Reinstall a particular package in all projects of the current solution:
Update-Package jQuery -reinstall
Update all packages in a project to the latest versions:
Update-Package -ProjectName MyProject
Reinstall all packages in a project:
Update-Package -ProjectName MyProject -reinstall
Update all packages in all projects of the current solution to the latest versions:
Update-Package
Reinstall all packages in all projects of the current solution:
Update-Package -Reinstall

Phew, that’s a lot of different combinations; but they are quite intuitive and easy to remember. To keep the list small, I left out the -IgnoreDependencies and -IncludePrelease parameters but you can add either (or both) of them to any of the example above (except that you can’t use -Version and -Reinstall together, remember?) to adjust the behavior of the command.

Hope that helps. Happy New Year.

Creating custom package rules for your build

2012-10-23T00:00:00-07:00

When you build your NuGet packages using nuget.exe, you may have noticed that sometimes you get a warning about the created package. Here’s one example:

This warning is generated by one of NuGet’s package analysis rules. Think of these as the NuGet equivalent of an FxCop rule. When nuget.exe creates a package, it automatically executes the package analysis step which in turn invokes all the registered rules. These rules will, one by one, examine the built package and emit the warnings as you see. You can disable this analysis step by supplying the -NoPackageAnalysis parameter to the ‘pack’ command, although we don’t recommend it.

By default, nuget.exe includes a set of built-in rules. These are:

InitScriptNotUnderToolsRule: detect whether there is an init.ps1 file that is not placed directly under the tools directory.
InvalidFrameworkFolderRule: detect whether there is an unrecognized framework folder name.
MisplacedAssemblyRule: detect whether any assembly files are placed outside of the lib folder.
MisplacedScriptFileRule: detect whether any Powershell .ps1 files are placed outside of the tools folder.
MisplacedTransformFileRule: detect whether a transform file (.pp or .transform) is placed outside of the content folder.
MissingSummaryRule: detect whether the Summary attribute is missing when the Description attribute is longer than 300 characters.
WinRTNameIsObsoleteRule: detect the presence of a ‘WinRT’ framework folder. (Starting with NuGet 2.1, the ‘WinRT’ moniker is obsolete and we recommend people use ‘NetCore45’ or ‘Windows8’ instead.)

What is not very well-known is that the rule set is extensible, meaning that you can write your own custom rules to cater to your needs. For example, the Microsoft ASP.NET team has written a rule to ensure that all packages that we ship have an Id that start with Microsoft.AspNet.. In this post, I will show you how easy it is to write a custom package analysis rule.

Develop rule assembly

First, create a class library project that targets .NET 4.0. You can add multiple rules in one assembly.
Add an assembly reference to nuget.exe. You need it for the package rule types. In the Properties window, set both Copy Local and Specific Version to False, because we don’t want to deploy nuget.exe together with our extension assembly.
Add another assembly reference to System.ComponentModel.Composition. You need this because the analysis mechanism is based on MEF.
Now add a class to your project for each rule you want to create. A rule class must derive from the NuGet.IPackageRule interface, and must export it using the MEF Export attribute.


[Export(typeof(IPackageRule))]
public class PackageIdRule : IPackageRule
{

}

Here’s the definition of the IPackageRule interface:


public interface IPackageRule
{
    IEnumerable<PackageIssue> Validate(IPackage package);
}

As you can see, it has only one simple Validate method, which receives an instance of the package that has just been built. The method will examine the package and return all the issues that it finds. An issue is represented by an instance of the PackageIssue class.


public class PackageIssue
{
    public PackageIssue(string title, string description, string solution);
    public PackageIssue(string title, string description, string solution, PackageIssueLevel level);

    public string Description { get; }
    public PackageIssueLevel Level { get; }
    public string Solution { get; }
    public string Title { get; }
}

Each issue has a Title, a Description, a Level, which can be either Warning or Error, and a proposed Solution. Note: currently, nuget.exe ignores the Level property and reports all issues as Warning.

Here’s the Validate method that we wrote for the previously mentioned rule to enforce all package Ids start with Microsoft.AspNet..


public IEnumerable<PackageIssue> Validate(IPackage package)
{
    if (!package.Id.StartsWith("Microsoft.AspNet.", StringComparison.Ordinal))
    {
        yield return new PackageIssue(
            title: "Package Id is not valid",
            description: "The Id of this package doesn't start with 'Microsoft.AspNet.*'",
            solution: "Rename the Id attribute so that it starts with 'Microsoft.AspNet.*'");
    }
}

Deploy rule assembly

After you’ve successfully compiled the project, you need to put the assembly into the right place. nuget.exe looks for extension assemblies in three places:

The %localappdata%\nuget\commands directory.
The current directory from which nuget.exe runs. Important: In this case, your assembly’s name must end with Extensions.dll, or else nuget.exe will ignore it.
A list of directories as specified by the NUGET_EXTENSIONS_PATH environment variable, separated by commas.

Summary

When nuget.exe builds a package, it runs a set of analysis rules to spot potential issues with the package. You can write your own custom rules to enforce additional requirements in your organization. A rule is simply a class that drives from the IPackageRule interface and is exported via MEF.

Invoking NuGet Services from inside Visual Studio

2012-09-26T00:00:00-07:00

One of the best kept secrets of the NuGet extension is that besides the UI that everyone has become familiar with, it also exports some useful services, via MEF, which allow other components in VS to interact with NuGet. These include installing packages, uninstalling packages and querying for installed packages.

Over the past year, the NuGet team has received many inquiries from various teams inside and outside Microsoft asking us how to perform such tasks from their components. We have solidified the guidance over time and wanted to share it broadly.

As of the current version (2.0), NuGet exports six interfaces/services via MEF, all of which reside in the NuGet.VisualStudio namespace in the NuGet.VisualStudio.dll assembly:

IVsPackageInstaller - Contains methods to install NuGet packages into projects.
IVsPackageInstallerEvents - Offers events which raise when packages are installed and uninstalled in the current solution.
IVsPackageInstallerServices - Contains a method to retrieve installed packages in the current solution, as well as methods to check if a specified package Id is installed in a project.
IVsPackageUninstaller - Contains methods to uninstall NuGet packages from projects.
IConsoleInitializer - Trigger eager initialization of the NuGet Package Manager Console.
IVsTemplateWizard - This interface is designed for project/item templates to include pre-installed packages, and is not meant to be invoked from code.

Here I will not discuss the detailed APIs in each of the interfaces. You can easily explore those for yourself in the Object Browser window.

Instead I will show you a step-by-step guide on how to reference the interfaces and use them in your code.

First, install the NuGet.VisualStudio nuget package into your project

The NuGet.VisualStudio package contains the NuGet.VisualStudio.dll assembly which includes all the above interfaces.

When installed, the package will automatically set the Embed Interop Types property of the assembly reference to True. The reason it does so is to make your code resilient against version changes when users update to newer versions of NuGet.

For the same reason, you must NOT use any other types besides the above interfaces in your code. You must NOT reference any other NuGet assemblies either, including NuGet.Core.dll.
Obtain the services

With the package installed, you are ready to obtain those services from your code. Because they are exported as MEF contracts, you can import them either via MEF’s Import attribute, or through the IComponentModel service in code. Here’s a sample code snippet:

var componentModel = (IComponentModel)GetService(typeof(SComponentModel));
IVsPackageInstallerServices installerServices = componentModel.GetService<IVsPackageInstallerServices>();
var installedPackages = installerServices.GetInstalledPackages();

So there you have it. If you find yourself needing to do something that the above services do not provide, feel free to open a discussion on the NuGet discussions board.

Recent Changes to Package Restore Consent

2012-09-07T00:00:00-07:00

When we released NuGet 2.0, one of the changes included the addition of a privacy-related constraint to the package restore feature. More specifically, NuGet 2.0 required that you provide explicit consent, via either a checkbox in the package manager configuration dialog or an environment variable, before it would successfully download required packages as a part of the package restore workflow.

The constraint was added specifically with build scenarios in mind. However, the implementation extended to other scenarios and as a result, added some unhappiness to the overall NuGet experience. Therefore, we’ve revisited the implementation and limited its impact to only the scenario that it was designed to address and none other.

The following changes have been made to the package restore workflow.

Added a ‘-RequireConsent’ command line flag to nuget.exe that, when present, causes NuGet to check for consent (this is the default behavior in NuGet 2.0)
Modified the NuGet targets file which is created by enabling package restore such that it passes the flag to nuget.exe when restoring packages during build

The impact of this change is twofold.

The default package restore workflow will continue to check for consent as it does today
Calling nuget.exe will no longer check for consent unless you explicitly pass the flag to the executable

The hope here is that while we need to continue checking for consent in the build scenario (since downloading is not a known or obvious build step), this change will eliminate any additional frustrations related to requiring consent when simply using nuget.exe by itself.

Additionally, because of the mechanics behind package restore, you won’t have to wait until the next NuGet release to take advantage of these changes. They are already available today as a part of the nuget.exe CodePlex download and the NuGet.Bootstrapper package. The act of enabling package restore for a solution will cause the latest version of nuget.exe to be downloaded automatically, so no further action is needed in this scenario. If you have an existing solution where package restore is already enabled, you can update it by doing the following:

Remove the .nuget folder from the solution in Visual Studio
Delete the .nuget folder from disk (keep Visual Studio open)
Right-click on the solution in the Visual Studio solution explorer and re-enable NuGet package restore

The process of enabling package restore will then run again, which will in turn download the latest version of nuget.exe and nuget.targets. Also, if you happened to close Visual Studio before deleting the .nuget folder from disk, you may notice that your projects fail to load when re-opening the solution. In this case, simply right-click on the solution and re-enable NuGet package explorer, then right-click on each of the failed projects and reload them.

Feed Performance Update

2012-08-24T09:30:00-07:00

As you might know, NuGet has been having some performance (and timeout) related issues recently. Earlier this week, we completed a deployment that helped, but it didn’t address everything.

Many users are still seeing slow responses or even timeouts when trying to use the ‘Manage NuGet Packages’ dialog in Visual Studio.

Ongoing Investigation

The deployment earlier this week greatly improved the packages page on the gallery, but it didn’t address the Visual Studio dialog performance as much as we had hoped. Since that deployment, we’ve been focusing on the queries behind the Visual Studio dialog.

We have found that the SQL queries that get executed from the ‘Manage NuGet Package’ dialog are not great. While the execution plans look okay, the memory grants for the queries are HUGE–MUCH bigger than they need to be. Because of the huge memory grants, the queries are stacking up behind each other waiting for memory to be granted. This is leading to poor performance and also timeouts.

We are working to get a different approach for these queries in place. We will let you know when a fix is ready. If you’re curious about what the queries look like, you can see the current and tentative rewritten queries here. The rewritten queries are using about 1/100th of the memory of the current queries.

To stay in touch with us, follow @nuget, or the #nuget tag on twitter for live updates. You can also check in on JabbR’s #nuget room.

Upgrade Successful

2012-08-22T23:30:00-07:00

Earlier this evening, we announced that the NuGet Gallery would be offline for about an hour during a scheduled upgrade. That upgrade is now complete.

We expect this deployment to help address performance issues we’ve been experiencing for the past week. We will continue to monitor the service and measure the results of this deployment.

CDN Re-Enabled

As a bonus, this deployment has re-enabled Windows Azure CDN for NuGet packages. We had initially enabled the CDN last week but some users around the globe reported the DNS name of the CDN host could not be resolved. After a week, we trust that the DNS name has propagated and we can now all benefit from faster package downloads.

Thank you once again for your patience. If you encounter any issues with NuGet, feel free to report issues on CodePlex.

To stay in touch with us, follow @nuget, or the #nuget tag on twitter for live updates. You can also check in on JabbR’s #nuget room.

Expected Downtime - August 22 11pm-12am PDT

2012-08-22T18:30:00-07:00

As mentioned in the previous post, we’ve been working to address some performance problems on the NuGet Gallery.

The fixes we have in place include some time-consuming database migrations. During our dry run deployments, we found that virtually all requests to the database will time out while these migrations are running. We have decided to take the precaution of shutting the gallery down during these migrations to ensure the best and fastest possible deployment. We are sorry for the inconvenience this will cause.

Timing

The nuget.org website is expected to be down from 11pm-12am PDT tonight. During this time, all requests to browse, search, install, or update packages will fail. You will see errors in Visual Studio when you use the Online tab in the Manage NuGet Packages dialog.

Live Updates

Please follow @nuget, or the #nuget tag on twitter for live updates. You can also check in on JabbR’s #nuget room.

This blog will be updated once the deployment is completed.

Workarounds

Many users have found Scott Hanselman’s tips on How to access NuGet when NuGet.org is down (or you’re on a plane) to be helpful.

Deployment Process

During this downtime, we will be doing the following:

Creating a pre-migration database backup
Executing the database migrations against the database
Performing a post-migration database backup
Deploying changes to the gallery web application
Verifying the successful deployment within a Staging instance of our Web Role

Once we have verified the successful staging deployment, we will swap the instances over to production. At that time service will resume.

If you’re curious about what is getting deployed, feel free to check out the commit history on the master branch in our GitHub repository at https://github.com/NuGet/NuGetGallery. The changes being deployed are from August 21st and 22nd.

Feature Request

Out of this situation comes a new feature request for the gallery: read-only mode. Ideally, deployments of this type could be handled by putting the gallery into a read-only mode and performing the deployment to a new web role with a copy of the database while the read-only gallery stays online. We will invest in setting up that model to be used in the future.

NuGet Gallery Performance Issues

2012-08-22T15:00:00-07:00

Since August 16th, and especially since August 21st, the NuGet Gallery has been exhbiting intermittent performance issues. Some users are reporting errors when attempting to browse or search for NuGet packages from Visual Studio or other clients. Many users are reporting that browsing and searching are slow. There have also been timeout errors when browsing the gallery’s packages page.

During this time, we have been communicating with many of you on twitter, under the #nuget tag, as well as in JabbR’s #nuget room. We truly appreciate your patience and understanding with us.

Status

We have been working to address these performance issues over the last several days, and we hope to have a deployment in place this evening (August 22nd, Pacific Time). It looks like the deployment will cause a short outage though–we will post more information about that when the details are available.

Root Cause

The root cause of the performance issues boils down to some sub-optimal queries that are used when searching the gallery–especially when searching from Visual Studio. There are a couple reasons these queries are slow:

Some missing indexes that could significantly aide the search queries.
Table columns that have nvarchar(max) as their data type when a size constraint can be applied. We are seeing that when these nvarchar(max) columns are used in ORDER BY clauses, it becomes a bottleneck.

The deployment we’re preparing will address both of these issues by adding size constraints to many fields and then creating several new indexes. Unfortunately, the process of altering the Packages and PackageRegistrations tables and then creating these new indexes is going to take some time to apply – during that time, the database will be rather unresponsive. Again, when we know more about the timing of the outage, we’ll share the details.

Results of NuGet CDN Experiment

2012-08-10T00:00:00-07:00

A little over a week ago, we asked for your help with an experimentt that we were conducting to see whether there was value in enabling CDN support for our packages blob storage container. Over 100 of you responded from all over the world, and the data that you provided made it pretty clear that there was sufficient justification for turning on CDN support for the production NuGet blob storage container. So firstly, thank you to everyone who participated in the experiment!

In addition to turning on CDN, we wanted to share the results of the data that you provided us. First, the worldwide numbers look like the following:

	No-CDN	CDN
Mean	33.8252	22.88007
Variance	601.3293	440.8645
Observations	108	108
Pearson Correlation	0.704394
Hypothesized Mean Difference	6.8
df	107
t Stat	2.420113
P(T<=t) one-tail	0.0086
t Critical one-tail	1.659219
P(T<=t) two-tail	0.0172
t Critical two-tail	1.982383

As you can see, there are some additional tests that we included so that you statisticians out there can get a deeper sense of our methodology, but at a high level, you can see that the worldwide average time to download the top 25 NuGet packages improved by over 10 seconds!

The improvements are even more pronounced depending on the specific region being considered. In the chart below, each vertical bar represents the range of values for the percentage of improvement and the horizontal line dividing each bar represents the average improvement. As you can see, the greatest improvements were seen in Australia with an average improvement of over 40%. Additionally, notable improvements can also be seen in each of the other regions from which we collected data.

You can anecdotally draw further conclusions to explain the latency numbers by looking at the landscape of Azure CDN nodes as it exists today.

All in all, the experiment was a great success, and we will be activating CDN in production very soon. We couldn’t have collected this level of data without your help, so again, many thanks!

Help Us Test CDN for NuGet.org

2012-08-02T00:00:00-07:00

Over the past few weeks, we’ve been investigating whether to turn on the content delivery network (CDN) feature of the Azure blob storage container for NuGet packages. In theory, this would make package downloads faster – especially if you’re located outside of the United States.

We want your help to conduct an experiment to measure the difference in downloading packages with CDN enabled verses disabled. Our initial experiment deployed nuget.exe into different regions and measured the time to download a set of NuGet packages. Unfortunately, in all of the regions we deployed to, we were still sitting on an Internet backbone connection, so even our baseline measurements (with CDN disabled) showed a statistically insignificant difference between a deployment in northern Europe and a deployment in the USA. As such, it would seem that enabling CDN wouldn’t yield any measurable benefit.

We think that the reason for the inconclusive baseline is due to the fact that there’s no “last mile” involved in the measurements, and this is where the majority of the slowdown happens. While we could try and simulate last mile time, we would rather change the experiment a bit and ask for your help.

We’ve created a custom of NuGet.exe along with a batch file runner that you can download here. Just unpack the zip file and run NuGet-CDN.bat. This will download the top 25 NuGet packages from both the standard package URL and a CDN URL and then log the time it takes to perform the download operations. You will then be asked to optionally provide your location, which will help us get a better sense of the global impact of using the CDN.

The batch process will wrap the log files along with your location information into a new NuGet package. Please email that package to nugetgallery@outercurve.org.

Thanks in advance for your help!

NuGet 2.0 Released

2012-06-19T00:00:00-07:00

We’re happy to announce that we released NuGet 2.0 on 6/19/2012. This release includes support for grouping dependencies, tools and content by the target framework of the project. Additionally, we’ve dramatically improved the performance of tab completion in the package management console.

Package restore consent is now active

As described in an earlier post on package restore consent, NuGet 2.0 will now require that consent be given to enable package restore to go online to download packages. Please ensure that you have provided consent via either the package manager configuration dialog or the EnableNuGetPackageRestore environment variable.

More details on NuGet 2.0 can be found on the release notes. Finally, NuGet 2.0 fixed several bugs. For a full list of work items fixed in NuGet 2.0, please view the NuGet Issue Tracker for this release.

NuGet 1.8 Hotfix Released

2012-05-29T00:00:00-07:00

With the NuGet 1.8 release, we found a breaking change that caused NuGet to fail on Windows XP machines. Earlier today, we released a hotfix to NuGet 1.8 that addresses this.

We apologize for any inconvenience this issue may have caused. If you have further issues, please report them on CodePlex.

For more information, there is a CodePlex discussion thread for this issue, which resulted in a work item to fix the regression.

NuGet 1.8 Released

2012-05-25T00:00:00-07:00

We’re happy to announce that we released NuGet 1.8 on 5/23/2012. This release includes support for localized satellite assemblies, nuget.exe performance improvements, and 34 bug fixes. Details are in the NuGet 1.8 Release Notes.

Known issue on Windows XP

We would like to call out that shortly after NuGet 1.8 was released, we learned that a cryptography change in 1.8 broke users on Windows XP. Do not install NuGet 1.8 onto a Windows XP machine; if you have already installed NuGet 1.8, you can uninstall it and reinstall NuGet 1.7 using this download link.

We are working on a NuGet 1.8.1 release to address the regression. There is a CodePlex discussion thread for this issue, which resulted in a work item to fix the regression.

Package Restore and Consent

2012-05-18T00:00:00-07:00

A little while back, our team had the pleasure of talking about NuGet with some privacy experts. These are folks who are passionate about your privacy and laws that protect your privacy.

As we were describing NuGet’s package restore feature, the privacy experts became interested in the details surrounding the package restore workflow – particularly the part where NuGet goes online to the NuGet gallery to fetch missing packages. Here’s the scenario that raised their interest:

Developer A creates a project, installs NuGet packages, and enables package restore
Developer A checks the project into version control. The NuGet packages aren’t included because of the package restore setting
Developer B checks the project out of source control and builds
Because package restore is enabled, NuGet goes online to the gallery, downloads and installs the required packages, and continues with the build.

That last point is where the concern was raised. In this scenario, Developer B has never said to NuGet, “Sure, you can go online on my behalf.” This can be a big deal because the simple act of making a request over the Internet can reveal information about the user (for example, from the user’s IP address, we can approximate her location). In some regions and countries, revealing this kind of information is a big no-no.

Now, none of us are lawyers and certainly not the type that work in international privacy issues. We just want to build a great product that developers can feel confident using, regardless of where they are. Hence the package restore consent feature.

What did we do exactly?

Our goal is that when someone builds a project that uses NuGet package restore, the person overseeing the build provides explicit consent to enable package restore to go online to download NuGet packages.

You’ll note that I was specific about who needs to provide the consent. When we were initially brainstorming solutions, we had talked about simply writing the consent flag into the NuGet config file or into build targets. However, these options would enable Developer A to effectively consent on behalf of Developer B, because these files are pulled down from version control, and that would defeat the purpose of requiring explicit consent. Therefore, we’ve provided 2 ways to specify package restore consent. The first approach is to check the box in NuGet’s package manager configuration dialog box as follows:

Checking this box will write the following XML into your NuGet.Config file (in your user profile directory tree)

<packageRestore>
  <add key="enabled" value="true" />
</packageRestore>

For scenarios where developers are building on their own machines, this will be the way to provide consent for package restore. However, there are cases where it’s not actually a person who’s doing the building and who therefore can’t provide consent this way. (And where Visual Studio isn’t even installed.) The prototypical example is a build server. In that case, NuGet will also look for an environment variable called EnableNuGetPackageRestore. To enable package restore for scenarios where the Visual Studio option is not practical, set this variable to true.

What if I haven’t given consent?

If you try to build a project that uses package restore and you haven’t given consent to NuGet to download packages, you will get a build error that looks like the following:

The build error describes the issue and provides instructions for providing consent. As the package restore consent feature rolls out, you will probably see this error, particularly on build servers, so we wanted to give you advance warning in this post.

It’s coming

In NuGet 1.8, we’ve added all of the logic for supporting the consent requirement. However, we won’t start actively enforcing the constraint (that is, causing build failures) until NuGet 2.0. This delay gives you time to provide consent for both your developer machines and your build infrastructure, and will help avoid sudden and unexpected build breaks when we release NuGet 2.0.

NuGet's Sources Are Moving to Git

2012-04-19T00:00:00-07:00

At the end of April, we’ll be moving the NuGet sources hosted on CodePlex from a Mercurial repository to a Git repository. This won’t impact or affect NuGet users, but NuGet contributors and those that follow NuGet’s development might need to act before this move.

Forks

All of the current NuGet forks on CodePlex will be deleted as part of this move. If you have forked NuGet, and you need to keep that fork, you can clone the fork and keep it locally or you can find somewhere else to push it. Aren’t DVCS great?

Pull Requests

The NuGet team will review all the currently active pull requests before we move to Git. We will not be accepting any new pull requests until this move is complete. Please wait to initiate new pull requests until that time.

Commit History

NuGet’s commit history will be preserved. If you’ve contributed to NuGet, your contributions will still be included in the commit history.

CodePlex Project

All other areas of the NuGet CodePlex project (the URL, issues, discussions, etc.) will remain as they are.

If you have questions or concerns, please present them in the comments or start a discussion on CodePlex.

NuGet 1.7 Released

2012-04-04T00:00:00-07:00

We’re happy to announce that we released NuGet 1.7 today. This release includes several new features as well as over 75 bug fixes. Details are in the NuGet 1.7 Release Notes.

The NuGet Gallery Outage on March 9th

2012-03-14T00:00:00-07:00

The NuGet Gallery was unavailable on March 9th from approximately midnight until 7:52 A.M., nearly eight hours. During this time all HTTP requests to the gallery website failed. In addition to affecting website users, the outage affected all NuGet client programs (such as the Visual Studio extension and nuget.exe) and other software that relies on the NuGet Gallery’s feed and API.

To start, we apologize for the disruption and inconvenience that this incident caused. We know that many of our customers were affected by the outage and we want to be transparent about what happened, what issues we found, how we plan to address these issues, and how we are learning from the incident to prevent a similar occurrence in the future.

The outage was caused by a NuGet Gallery application error, compounded by some shortcomings in our monitoring system and incident-response process for the NuGet gallery.

What Happened Exactly

Windows Azure rebooted the NuGet Gallery servers at midnight after performing a scheduled OS upgrade. After this reboot, the gallery’s web roles failed to initialize due to a previously unknown bug in a script we wrote that we execute in order to enable HTTP compression for the gallery’s feed. Because the web roles could not initialize, the website became unresponsive. Our monitoring service didn’t send any alerts about this situation. However, the team became aware of the problem around 7:30 A.M. and we were able to restore service by deleting the Azure deployment and creating a new one (bypassing the scripts’s bug, although we didn’t know about the bug yet).

How We Failed

As with most system failures of this severity, there was more than just one failure.

Deployment Testing

We continually and automatically test the process of deploying new versions of the gallery to Windows Azure by using a preview site. However, we missed testing reboot scenarios, and this masked a bug in our compression script.

Monitoring Service

As a result of this outage, we learned that our monitoring service had configuration issues that we did not know about. We understand that it’s crucially important to continually test all of the systems that keep our website running, including things like monitoring, backup, and restore. We test our backup and restore continually using our preview site, but as we’ve learned, we were not testing monitoring sufficiently.

Inadequate Incident-Response Process

When the outage occurred, we didn’t have a robust enough incident-response process, with the result that the response time for this outage that was simply not fast enough. There has been no dedicated operations team for the NuGet Gallery, and the development team has been assuming all responsibility for operational activities.

Bad Assumptions

One of the reasons we had not formalized our incident response process or had people on call was that the NuGet gallery has been very reliable. (As we’ve now been reminded, this is no excuse for lack of proper planning.)

We have since also learned that two assumptions about how Windows Azure deployments work contributed to this issue. First, we assumed that we would be notified that an OS upgrade requiring a reboot had been scheduled. We’ve learned that Windows Azure does not currently send notifications for scheduled OS upgrades, whether a reboot is required or not. We’ve recommended to the Azure team that they start sending notifications in these cases.

Second, we assumed that OS upgrades (or other maintenance) would be performed on one instance at a time within a deployment, and that if an upgrade failed, Azure would not then try to upgrade the remaining instances, effectively causing the entire deployment to fail. This one-at-a-time approach is how Azure handles customer-initiated upgrades. We’ve recommended to the Azure team that they consider changing the Azure-initiated upgrade process accordingly, although we understand there are a lot challenges in doing this.

What We’re Changing

All applications can fail, which is why it’s crucial to have a proven, tested system in place to monitor the health of an application, and a proven, tested process in place to respond to incidents. To that end, we will:

Regularly test our monitoring service (for example, by periodically making the preview site unavailable).
Create a better incident-response process.
Expand the team of people who understand Azure deployment and know how to troubleshoot failures.
Always have at least one person on call to respond to incidents.

We will also regularly test server reboots after a deployment, using our preview site.

While we can’t guarantee that future incidents won’t happen (in fact, we know they will), we can guarantee that our response to any future incidents will be much more timely.

Introducing the NuGet Team Blog

2012-03-13T00:00:00-07:00

For the last few months, the NuGet team has been considering starting a proper, official blog. This is that blog.

How do we plan to use this blog? Here are a few ways we know we’ll use it (and I’m sure we’ll think of others as we go along):

Announcing new releases
Announcing upcoming breaking or very impactful changes
Pointing out particuarly important discussions on our CodePlex or GitHub sites

Feel free to leave us a comment, letting us know what else you’d like to see here.

The NuGet Team Blog

Where else is NuGet used in Visual Studio?

Play with Packages, programmatically!

Getting Started

Get me the pre-released versions of a package

Help me get a package

It’s quite easy, have fun!

A quick note about the new MinClientVersion attribute

Support for Native Projects

Visual Studio Support

MSBuild Integration

Native Target Framework

Creating Native Packages

Feedback

Related Links and Announcements

NuGet 2.5 Released

Download it now

Acknowledgements

What’s Included in NuGet 2.5

Here are the most notable features:

NuGet’s description

XDT web.config transforms

Goodbye preview.nuget.org. Hello staging.nuget.org!

Goodbye preview.nuget.org

Hello staging.nuget.org

NuGet 2.5 Release Candidate

What’s Included in NuGet 2.5

Here are the most notable features:

NuGet’s description and slogan

Recent NuGet Gallery Updates

April 4, 2013

Top 500 packages exposed in the feed

Numeric rank for packages stats

Links to gravatar in profile page

UserName optimization in DB (backend)

Other minor bug fixes

March 28, 2013

Support for MinClientVersion

Contacting support

Improved package statistics

Other minor bug fixes

The March 6th NuGet Gallery Outage

Timeline of events

The Problematic Code Changes

Avoiding the database for package downloads

The Postmortem - What Went Wrong

Conclusions - What We Plan to Change

Improved search syntax on nuget.org and in NuGet Visual Studio extension

Introducing Package Statistics

Statistics Tab

Downloaded Packages

Downloaded Package Versions

Version Breakdown for a Package

Statistics for any Package

Availability

Feedback and Input

Workflow Changes for CodePlex Issues

Managing Package Owners on nuget.org

Setting a Package’s Initial Owner

Specifying Additional Package Owners

Removing Package Owners

Transfering Package Ownership

Orphaned Packages

A quick tutorial on the Update-Package command

Creating custom package rules for your build

Develop rule assembly

Deploy rule assembly

Summary

Invoking NuGet Services from inside Visual Studio

First, install the NuGet.VisualStudio nuget package into your project

Obtain the services

Recent Changes to Package Restore Consent

Feed Performance Update

Ongoing Investigation

Upgrade Successful

CDN Re-Enabled

Expected Downtime - August 22 11pm-12am PDT

Timing

Live Updates

Workarounds