Consolidated REST API deployed

A few weeks back, we deployed our consolidated REST API powering NuGet.org and the NuGet client experience in Visual Studio. An invisible change for our users, but a big change for the NuGet team. This consolidated REST API lays the foundation of our future work on the server side of NuGet. In this post, we want to expand a little on the history of the NuGet services and why it matters for us.

History of our internal REST API

Historically, NuGet.org used to serve all requests for packages directly from our database. With the number of NuGet users increasing, the number of queries on our database also increased. Around two years ago, the NuGet team built an internal REST API for use by our own services. It would provide package metadata, provide search results, handle download counts and all that - with great results: the database got some breathing room.

For the beta release of the NuGet 3.0 client, we faced a difficult decision: should we expose this REST API to the public, or build a new one? We decided to build a new, public REST API that would be used by our future NuGet clients. This decision was made because of several reasons. First, we wanted to move on from our OData API that was used in previous NuGet clients. Second, the REST API we had in place was very opinionated and tailored to our own services and not well suited for public access by our users.

So we ended up with an internal and an external REST API. When new features were being added in NuGet 3.0 and beyond, we had to implement them in all of our API’s - resulting in a lot of duplicate work. Both API’s had their own caching layer, which proved quite the challenge in keeping data in sync.

What we did

A few months ago, we set on an adventure of merging both REST API codebases and refactoring them to use the same caching layer. We cleaned up a lot of redundant code paths and optimized the caching layer. While not the scope of this effort, we also introduced a few minor improvements to several features - for example, we now recognize several commonly used acronyms like “EF” (Entity Framework) or “UWP” (Universal Windows Platform) in our search engine.

We fixed several tokenization bugs and differences between the two services. A large test suite for search was added as well, boosting our confidence in future changes and investment in the search engine.

Some improvements were introduced to the search results, like boosting on exact match to package id. A new logarithmic scale for download counts ranks packages with a high number of downloads slightly better - an algorithm which we will be tweaking further.

A lot of work has been done on performance. We profiled the API during load tests to optimize code paths and reduce the API’s memory traffic and garbage collector pauses. One notable improvement is the cold start of the application, which is down to a minute as opposed to the original 7 minutes.

Every few weeks we had a “flighting” moment where we would deploy our changes to a production-like environment, serving production traffic. This allowed us to benchmark the consolidated REST API and verify its responses.

This week, we deployed this consolidated REST API. And nobody noticed - the very nature of this sort of change. We are really excited about it. Reduced maintenance is a benefit we’re really happy with. The refactored code base also lays the foundation for future work on our REST API.

What’s next?

The consolidated REST API, much like our switch from WCF OData to Web API OData, is a project to reduce maintenance and improve the foundation on which the NuGet services are built, for today and tomorrow. We are also working towards documenting the API.

We want to hear your feedback!

We want NuGet to meet the evolving needs of our community. If you would like share your pain points and your current and future needs, use the calendly link to set up a quick 15-30 min call with us. If you would like to send us an email instead, hit us up at feedback@nuget.org.

You can also leave a comment below, and as always, if you run into any issues or have an idea, open an issue on GitHub.

Published May 19, 2016 by Maarten Balliauw

The 1st Billion

Today, NuGet.org reached one billion downloads. This is a momentous achievement for our users and the community of package authors who continue to use and build new libraries that is the cornerstone of .NET adoption. We want to take this opportunity to give a huge thank you to the millions of our users who made this milestone possible. With the advent of .NET Core, ASP.NET Core, Universal Windows Apps, open sourcing of the Xamarin Platform, and the accelerated use of NuGet, we believe we will hit the next 1 billion in a year or so.

Journey to 1 Billion

NuGet started out as the NuPack project which was then renamed to NuGet and accepted into the ASP.NET Open Source Gallery by the OuterCurve foundation. This was one of the first forays by Microsoft into Open Source development and we are proud of that heritage. For a taste of nostalgia, here are some of the very first posts around the NuGet project: Scott Guthrie, Phil Haack, Scott Hanselman and Mary J Foley

Here are some fun stats around our growth over the last 6 years. Data from the past 6 weeks or so across some of these dimensions are available in our statistics page in NuGet.org.

Package Downloads

Downloads in NuGet.org grew almost 80% this year. From an all time download count of 50 million in 2013, we now average around 72 million downloads per month in 2016. NuGet from a day 1 was built with caching in mind and we have seen significant usage of NuGet via NuGet Server, VSTS Package Management, myget, ProGet, Nexus, Artifactory and other third party services. These download numbers purely represent downloads from NuGet.org. We suspect the actual downloads of NuGet packages to be a much higher number.

If you were curious which was the billion’th package that was downloaded. The magic number was taken by the dnx-clr-win-x86 package, which was downloaded at around 3:15 a.m (PST) on 5/10/2016.

Package Uploads

We have over 54K unique packages in NuGet.org and over 600K packages in total. Year over year we are seeing around 31% growth in the number of packages being uploaded to NuGet.org.

Package Uploads

Tools Usage

This visualization shows the average count of requests from NuGet v3 client, the Windows x86 Command line and the .NET CLI NuGet Client over the last 2 weeks to NuGet.org endpoints. As you can see, there is huge interest in .NET Cross platform tools and we believe this is going to be a significant part of our tools traffic going forward.

Tools Usage

NuGet.org

We now have around 600K users visiting NuGet.org on a monthly basis to search and peruse our packages. We are making investments in improving search and package information in the near future to make user engagement much more meaningful on the website.

Site Usage

Community

An Open Source project thrives and survives on the patronage of its users. Even from the very early days, we have had significant contributions from the community. 3 out of the top ten contributors in the initial days were individuals from the community. Fast forwarding to today, we have over 130 contributors on GitHub across NuGet.Client and NuGetGallery. While a number of these folks are MSFT’s, there is a large number of contributions from the community as well. We are always on the lookout for folks from the community to suggest ideas, and contribute features to the project.

Among non-Microsoft packages. Json.net authored by our friend James Newton-King reigns supreme with over 28 million downloads. Our most prolific author/owner with the largest number of packages at this time is Definitely Typed which curates high quality TypeScript definitions with over 1,779 packages.

We cannot reiterate enough how much our community means to us and we want to continue to partner with you all to innovate and push the boundaries of package management with NuGet.

Why you should be using NuGet

There is a lot of innovation happening in the community around .NET which enables you to develop software faster than ever. NuGet is the most widely adopted system across the community for consuming shared components and is the primarily channel for showcasing this innovation. One of the big goals for us is to optimize developer productivity, reduce redundancy and promote code-reuse. Using NuGet Packages is one the most optimal ways to do this. We are here to help you succeed in adopting NuGet and if you would like to talk to us about how NuGet can help you achieve these goals, do not hesitate to reach out to us at feedback@nuget.org or holler at us on Twitter @nuget.

We want to hear your feedback!

We want NuGet to meet the evolving needs of our community. If you would like share your pain points and your current and future needs, use the calendly link to set up a quick 15-30 min call with us. If you would like to send us an email instead, hit us up at feedback@nuget.org. You can also leave a comment below, and as always, if you run into any issues or have an idea, open an issue on GitHub.

Published May 10, 2016 by Harikrishna Menon

Introducing The NuGet Beta Channel

Today, we would like to introduce you to the NuGet Beta Channel for the Visual Studio 2015 NuGet Package Manager. In the last 2 months, we have been focusing on improving the quality of the package management experience in Visual Studio 2015 and we have released over 3 consecutive versions of the package manager with added features, improved quality and performance. We want to able to accelerate this cadence and make sure that our customers are able to access new features and bug fixes faster.

We have been trying to accelerate our releases, but we realized that even when we share beta bits, it takes too much effort to discover them, and we end up getting bug reports only after we release the RTM version. We would like to reach interested users with our beta releases for feedback. This way we can simultaneously share improvements faster, and at the same time impact a smaller ring of users if we end up regressing a scenario.

Who is this for?

The beta channel is recommended for the following users: -

  • You like to stay on the latest and greatest.
  • You want to try out new features.
  • You are experiencing a blocking issue and want to get access to the build with the fix before it hits RTM.

We also hope that you use this opportunity to give us feedback on the quality of the release. If you run into any issues whatsoever, feel free to open an issue on GitHub.

What goes into making a Beta build?

Beta fixes go through a decent amount of dedicated validation and dog-fooding by the team and our partners. Even though it’s called a beta, we will only release near RTM quality builds into this channel. We want to use the extra runway on our end to make sure that we incorporate any feedback that we might get from the users of the channel and catch any blocking issues early on.

How do I get access to the Beta Feed?

You can get access to the Beta builds by the following the steps outlined below. We have published the 3.4.4 RC build fixing these issues to this channel today.

  • Add the Beta Feed: https://dotnet.myget.org/F/nuget-beta/vsix/ to the Additional Extension Galleries list in Tools->Options->Environment->Extensions and Updates.

Extensions and Updates Settings

  • Navigate to Tools->Extensions and Updates and select Online. You should now be able to see the NuGet-Beta Feed there. Install the NuGet Package Manager Extension.

Extensions and Updates

  • You are all set to get beta releases! Remember if you run into any issues while dogfooding the beta build or have an idea, open an issue on GitHub.

We want to hear your feedback!

We want NuGet to meet the evolving needs of our community. If you would like share your pain points and your current and future needs, use the calendly link to set up a quick 15-30 min call with us. If you would like to send us an email instead, hit us up at feedback@nuget.org.

You can also leave a comment below, and as always, if you run into any issues or have an idea, open an issue on GitHub.

Published May 02, 2016 by Harikrishna Menon

Older Posts